A Method and System for Enhancing Question Answering in Long Videos Based on Bidirectional Audio-Visual Alignment

Abstract

This invention discloses a method and system for enhancing long video question answering based on bidirectional audio-visual alignment, aiming to solve the problems of missing keyframes in uniform frame extraction, audio-video semantic fragmentation, and the need to modify the model for multimodal fusion in existing long video question answering systems. It constructs a bidirectional alignment mechanism through audio processing and ASR construction, audio-to-visual A2V keyframe retrieval, and visual-to-audio V2A frame-level ASR injection and serialization, achieving precise temporal and semantic alignment of audio and video at the input layer. This solution does not require modification of the existing video language model backbone, has plug-and-play characteristics, and improves accuracy by 3.6% and 2.2% on Video-MME and LVBench respectively, effectively improving the inference accuracy of long video question answering and is applicable to various long video scenarios.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal information processing technology, specifically to a long video question-answering enhancement processing method and system based on bidirectional audio-visual alignment. Background Technology

[0002] With the development of Multimodal Large Models (MLLM), Video Question Answering (VideoQA) has become a major research direction in the field of video understanding. Current mainstream methods convert videos into image frames by uniformly sampling the frames, and then input them into the model for inference along with the text questions. This approach is effective in short videos or scenes with dense visual events, but in videos with long time sequences, sparse information, or where the main events occur only briefly, it is very easy to miss key evidence frames, leading to inference failure.

[0003] Audio information such as speech and dialogue in real videos carries high-density semantics and clear temporal references, which are crucial for understanding videos. However, existing methods either completely ignore audio or only use the complete ASR text as global context input, failing to establish a precise correspondence between audio semantics and specific visual moments, resulting in a temporal and semantic disconnect between audio and video.

[0004] Some studies have attempted to achieve joint audio-video modeling through dedicated audio encoders or cross-modal fusion networks. However, such methods require modification of the original model structure or retraining, resulting in high engineering costs, poor generalization, and unstable audio-video feature alignment. Therefore, there is an urgent need for a general enhancement method that can achieve accurate audio-video alignment at the input level without modifying the model backbone. Summary of the Invention

[0005] To address these issues, the present invention provides a long video question-answering enhancement processing method and system based on bidirectional audio-visual alignment, in order to solve the problems in the prior art, such as the omission of major visual events, failure of audio-visual collaborative reasoning, low accuracy, and difficulty in plug-and-play operation, caused by the difficulty in locating key local information due to uniform frame extraction, lack of precise alignment between audio semantics and visual moments, and the need to modify or retrain the model for multimodal fusion.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a long video question-answering enhancement processing method based on bidirectional audio-visual alignment, comprising the following steps: S1, Audio Processing and ASR Construction: Obtain the input video and corresponding audio stream, segment the audio using a sliding window approach, perform speech recognition on each audio segment, and integrate the transcription results of overlapping audio segments using a language model to generate... ASR Fragment collection; S2, Audio-to-Visual Alignment A2V For the input question text, calculate the relationship between the question text and the input question text. ASRThe semantic similarity of each audio text in the fragment set is used to filter out those relevant to the problem. ASR Fragments, in each of the selected... ASR Within the time interval corresponding to the segment, the video is intensively frame-by-frame extracted. Then, by calculating the visual-text similarity between the candidate frames and the question text, visual frames related to the question are selected to form a set of visual supplementary frames. S3, Visual-to-Audio Alignment V2A The input video is subjected to uniform frame extraction to obtain a base frame set. The base frame set is then merged with the visual supplementary frame set to obtain the final visual input set. Based on the timestamps of each visual frame in the final visual input set, the... ASR The speech text within the corresponding time interval is matched from the fragment set. The speech text is used as a local audio context aligned with the visual frame time to form a multimodal input unit. The multimodal input unit is serialized into a unified multimodal input representation accepted by the video language model according to a predetermined format. S4. Video Question Answering Reasoning: Input the multimodal input representation and question text into the video language model, and output the predicted answer to the question.

[0007] As a preferred embodiment of a long video question-answering enhancement processing method based on bidirectional audio-visual alignment, the time interval of the audio segment divided by the sliding window in S1 is: ,and Δ <W In the formula, The aforementioned set of audio segments, The audio segment, W For window length, Δ The window step size; In S1, speech recognition is performed on each audio segment to obtain the initial transcribed text:

[0008] In the formula, For the initial transcribed text, For the speech recognition; The generated in S1 ASR The expression for the fragment set is:

[0009] In the formula, For the purposes of this statement ASR Fragment collection, This is voice-to-text content. The start time corresponding to the spoken text. This is the end time corresponding to the spoken text.N For the ASR Total number of fragments The aforementioned voice text; The ASR The fragment set includes the speech text content and the corresponding start time and end time of the speech text.

[0010] As a preferred scheme for long video question-answering enhancement processing based on bidirectional audio-visual alignment, the question text in S2 and the... ASR The semantic similarity of audio-visual text segments is calculated as follows:

[0011] In the formula, The semantic similarity is mentioned above. For text semantic encoding functions, cos(·) is Cosine similarity calculation function, This refers to the content of the spoken text. Q The input is the question text.

[0012] As a preferred scheme for long video question-answering enhancement processing based on bidirectional audio-visual alignment, S2 filters out the relevant information in the question. ASR The number of segments is K , K The value of is 8, and the set expression is:

[0013] In the formula, For K of the aforementioned ASRs Fragment collection, The semantic similarity is mentioned above. for K ASR voice text Excerpt.

[0014] As a preferred scheme for long video question-answering enhancement processing based on bidirectional audio-visual alignment, the candidate visual frame set obtained by dense frame extraction in S2 is expressed as follows:

[0015] In the formula, For the ASR The set of candidate visual frames within the corresponding time interval of the segment. The candidate visual frames in this set, The aforementioned voice text. M This represents the total number of densely drawn frames within that time interval.

[0016] As a preferred scheme for long video question-answering enhancement processing based on bidirectional audio-visual alignment, the visual-text similarity calculation method in S2 is as follows:

[0017] In the formula, For visual encoding functions, For text semantic encoding functions, cos(·) is Cosine similarity calculation function The candidate visual frames in this set, Q For the input question text, For the ASR The set of candidate visual frames within the corresponding time interval of the segment. For the ASR The fragment corresponds to the visual frame in the candidate visual frame set that is most relevant to the question text.

[0018] As a preferred scheme for long video question-answering enhancement processing based on bidirectional audio-visual alignment, the expression for the visual supplementary frame set in S2 is:

[0019] In the formula, For the visual supplementary frame set, For the ASR The fragment corresponds to the visual frame in the candidate visual frame set that is most relevant to the question text. K For the ASR The number of segments.

[0020] As a preferred scheme for a long video question-answering enhancement processing method based on bidirectional audio-visual alignment, the expression for the base frame set obtained by uniform frame extraction in S3 is:

[0021] In the formula, For the basic frame set, For the first The aforementioned basic visual frames. For the first The timestamps corresponding to the basic visual frames. T The total number of frames in the uniform frame extraction, and T=32 ; The final visual input set is:

[0022] In the formula, For the final visual input set, This is the set of supplementary visual frames; The visual frame and the ASR The matching criteria for the fragment are: ,satisfy

[0023] In the formula, In order to be with the first The speech text matched by the visual frame. The spoken text content. For the first The timestamp of each of the visual frames. The start time corresponding to the spoken text. The end time corresponding to the spoken text.

[0024] As a preferred scheme for long video question-answering enhancement processing based on bidirectional audio-visual alignment, the multimodal input representation and the model output obtained by inputting the question text into the video language model in S4 are as follows:

[0025] In the formula, The predicted answer to the question text. For the video language model, This is the final multimodal input representation. Q is The text of the question entered.

[0026] This invention also provides a long video question-answering enhancement processing system based on bidirectional audio-visual alignment, employing the aforementioned long video question-answering enhancement processing method based on bidirectional audio-visual alignment, comprising the following steps: Audio processing and ASR The module is used to acquire the input video and its corresponding audio stream, segment the audio using a sliding window approach, perform speech recognition on each audio segment, and integrate the transcription results of overlapping audio segments using a language model to generate a... ASR Fragment collection; Audio to Visual Alignment A2V This module is used to calculate the relationship between the input question text and the given question text. ASR The semantic similarity of each audio text in the fragment set is used to filter out those relevant to the problem. ASR Fragments, in each of the selected... ASR Within the time interval corresponding to the segment, the video is intensively frame-by-frame extracted. Then, by calculating the visual-text similarity between the candidate frames and the question text, visual frames related to the question are selected to form a set of visual supplementary frames. Visual to audio alignment V2AThe module is used to perform uniform frame extraction on the input video to obtain a base frame set, merge the base frame set with the visual supplementary frame set to obtain a final visual input set, and, based on the timestamp of each visual frame in the final visual input set, perform... ASR The speech text within the corresponding time interval is matched from the fragment set. The speech text is used as a local audio context aligned with the visual frame time to form a multimodal input unit. The multimodal input unit is serialized into a unified multimodal input representation accepted by the video language model according to a predetermined format. The video question-answering reasoning module is used to input the multimodal input representation and question text into the video language model and output the predicted answer to the question. The present invention has the following advantages: First, this invention constructs an audio-to-visual interface. A2V The alignment mechanism uses audio semantics as a time navigation signal to accurately locate the audio time intervals related to the question and perform intensive frame extraction. This effectively makes up for the shortcomings of traditional uniform frame extraction, which is prone to missing key local events. It ensures that key visual evidence that appears briefly in long videos and information-sparse videos is not ignored, providing sufficient basis for model inference.

[0027] Second, this invention utilizes visual-to-audio... V2A Aligned frame level ASR The injection mechanism enables precise binding of each visual frame with the corresponding timestamp audio text, allowing the model to simultaneously acquire "visual images and speech semantics at the same moment" during inference. This completely solves the problem in existing methods where audio text is only used as global context and is disconnected from visual frame time and semantics, significantly improving the collaborative inference effect of cross-modal information.

[0028] Third, the present invention adopts an input layer optimization scheme, which does not require modification of the backbone structure and parameters of the existing video language model, nor does it require the introduction of an additional audio encoder or retraining. It has plug-and-play characteristics and can be directly adapted to a variety of mainstream video question answering models, which greatly reduces the engineering cost of technology implementation and is suitable for a variety of scenarios such as long videos, dialogue-intensive videos, and educational videos.

[0029] Fourth, this invention achieves enhanced collaboration through a bidirectional audio-visual alignment structure, enabling "audio-guided visual retrieval and visual regression to audio semantics," combined with... Qwen3-ASR-Flash Accurate transcription o3-mini Overlapping fragment integration and Qwen3- 3B semantic matching optimization, in Video-MME and LVBench In the two major video understanding tests, based on Qwen2.5-VL-7B The model's question-answering accuracy improved respectively 3.6% and 2.2%The technical effects can be quantified and verified.

[0030] Fifth, this invention not only solves the pain point of difficulty in locating key information in long videos, but also meets the needs of scenarios with dense dialogue and semantic dependence on audio supplementation. At the same time, its low adaptation cost and high reasoning accuracy make it a promising prospect for promotion and practical value in multiple fields such as intelligent interaction, video content retrieval, online education Q&A, and film and television content analysis. Attached Figure Description

[0031] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0032] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0033] Figure 1 This is a schematic diagram of a long video question-answering enhancement processing method based on bidirectional audio-visual alignment provided in an embodiment of the present invention; Figure 2 This is a technical architecture diagram of a long video question-answering enhancement processing method based on bidirectional audio-visual alignment provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a long video question-answering enhancement process based on bidirectional audio-visual alignment provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a long video question-answering enhancement processing system architecture based on bidirectional audio-visual alignment provided in an embodiment of the present invention. Detailed Implementation

[0034] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Example 1

[0036] See Figure 1 , Figure 2 and Figure 3 This invention provides a method for enhancing question-answering in long videos based on bidirectional audio-visual alignment, comprising the following steps: S1, Audio Processing and ASR Construction: Obtain the input video and corresponding audio stream, segment the audio using a sliding window approach, perform speech recognition on each audio segment, and integrate the transcription results of overlapping audio segments using a language model to generate... ASR Fragment collection; S2, Audio-to-Visual Alignment A2V For the input question text, calculate the relationship between the question text and the input question text. ASR The semantic similarity of each audio text in the fragment set is used to filter out those relevant to the problem. ASR Fragments, in each of the selected... ASR Within the time interval corresponding to the segment, the video is intensively frame-by-frame extracted. Then, by calculating the visual-text similarity between the candidate frames and the question text, visual frames related to the question are selected to form a set of visual supplementary frames. S3, Visual-to-Audio Alignment V2A The input video is subjected to uniform frame extraction to obtain a base frame set. The base frame set is then merged with the visual supplementary frame set to obtain the final visual input set. Based on the timestamps of each visual frame in the final visual input set, the... ASR The speech text within the corresponding time interval is matched from the fragment set. The speech text is used as a local audio context aligned with the visual frame time to form a multimodal input unit. The multimodal input unit is serialized into a unified multimodal input representation accepted by the video language model according to a predetermined format. S4. Video Question Answering Reasoning: Input the multimodal input representation and question text into the video language model, and output the predicted answer to the question.

[0037] In one possible embodiment, the time interval of the audio segment divided by the sliding window in S1 is: ,and Δ <W In the formula, The aforementioned set of audio segments, The audio segment, W For window length, Δ The window step size; Specifically, sliding window segmentation is used to avoid semantic loss due to truncation of speech content. Δ <W The parameter settings enable window overlap, ensuring speech integrity across segments.

[0038] Among them, window length W The time span of a single audio segment is determined and needs to be set according to the length of common speech segments (such as dialogues and narration) to ensure that a single recognition can completely cover a semantic segment; window step size Δ Less than window length W This allows adjacent audio segments to overlap, avoiding transcription breaks caused by speech content crossing window boundaries, ensuring semantic coherence of the speech text, and facilitating subsequent integration of overlapping segments and generation of accurate audio. ASR Sets provide the foundation.

[0039] In S1, speech recognition is performed on each audio segment to obtain the initial transcribed text:

[0040] In the formula, For the initial transcribed text, For the speech recognition; Specifically, speech recognition ASR Converting audio signals into text semantics solves the problem that audio information cannot be directly semantically associated with text or visual frames.

[0041] Among them, speech recognition ASR The model is the main tool for connecting audio signals with text semantics, by using the audio segments corresponding to each sliding window. Convert to initial transcribed text This enables the semantic transformation of audio information, allowing audio that could not originally participate directly in text semantic calculations to be matched with problem text and visual frames, providing a semantic bridge for cross-modal alignment.

[0042] The generated in S1 ASR The expression for the fragment set is:

[0043] In the formula, For the purposes of this statement ASR Fragment collection, This is voice-to-text content. The start time corresponding to the spoken text. This is the end time corresponding to the spoken text. N For the ASR Total number of fragments The aforementioned voice text; The ASRThe fragment set includes the speech text content and the corresponding start time and end time of the speech text.

[0044] Specifically, by integrating the transcription results of overlapping segments using a language model, text redundancy caused by sliding window segmentation can be eliminated, generating text that is both accurate and temporally relevant. ASR The collection of fragments provides a precise time navigation basis for the subsequent alignment logic of "from hearing to seeing".

[0045] This expression defines a structured ASR fragment set format, the main value of which lies in providing each speech text with a specific format. sj Binding to a precise time range [ The introduction of timestamps transforms speech text from an isolated global context into a precise match with the timestamps of video frames, facilitating subsequent processing. A2V The module's "frame extraction by audio time interval" and V2A The module's "match audio by visual frame timestamp" provides the main time correlation basis and is the main data foundation for achieving bidirectional alignment.

[0046] In one possible embodiment, the question text in S2 is related to the... ASR The semantic similarity of audio-visual text segments is calculated as follows:

[0047] In the formula, The semantic similarity is mentioned above. Here, cos(·) is the text semantic encoding function, and cos(·) is the cosine similarity calculation function. Q represents the content of the spoken text, and Q is the input question text.

[0048] Specifically, firstly, semantic similarity calculation is used to associate the text question with the audio semantics, quickly locating the audio time interval that may contain evidence of the answer (i.e., the time interval related to the question). ASR (Segments), to avoid indiscriminately traversing the entire long video.

[0049] Among them, the text semantic encoding function Its function is to translate the question text Q and voice text This transforms the text into a high-dimensional semantic feature vector, enabling the semantic information of the text to be quantified through vector space distance; the cosine similarity calculation function... cos(·) This is used to measure the angle between two semantic vectors; the smaller the angle, the higher the similarity. It can accurately quantify the semantic relevance between the question text and each spoken text, thereby filtering out those most likely to contain clues to the answer. ASR Fragments provide precise semantic guidance for visual keyframe retrieval.

[0050] In one possible embodiment, the filtering of the relevant questions in S2 ASR The number of segments is K , K The value of is 8, and the set expression is:

[0051] In the formula, For K of the aforementioned ASRs Fragment collection, The semantic similarity is mentioned above. for K ASR voice text Excerpt.

[0052] Specifically, K The choice of value is a balance between retrieval efficiency and coverage of key information: K A value that is too small may miss some audio segments that are relevant to the problem, resulting in key visual frames not being retrieved; K An excessively large value will introduce redundant audio segments, increasing the engineering cost of subsequent intensive frame extraction and similarity calculations. This is based on the test data provided in the supplementary documentation. K=8 At the same time, it can minimize computational overhead while ensuring that no key information is omitted. The filtering logic directly extracts the top results with the highest semantic similarity. K indivual ASR Fragments, forming a precise set of visual retrieval time intervals. .

[0053] In one possible embodiment, the candidate visual frame set obtained by the dense frame extraction in S2 is expressed as:

[0054] In the formula, For the ASR The set of candidate visual frames within the corresponding time interval of the segment. The candidate visual frames in this set, The aforementioned voice text. M This represents the total number of densely drawn frames within that time interval.

[0055] Specifically, intensive frame skipping within the relevant audio time interval is because the probability of visual events related to the problem occurring in this interval is significantly higher than in other areas, and intensive frame skipping can maximize the capture of the main scene.

[0056] Among them, in the selected ASR Dense frame skipping is performed within a specific time interval because that interval is highly relevant to the semantics of the problem. MA larger value can maximize the capture of visual details within that range, avoiding the omission of major visual events (such as briefly appearing actions or objects) due to excessively large frame skipping intervals; candidate visual frame set For each related ASR A "visual candidate pool" is established for the visual regions corresponding to the segments, providing a sufficient sample base for subsequent secondary selection of the optimal visual frames.

[0057] In one possible embodiment, the visual-text similarity in S2 is calculated as follows:

[0058] In the formula, For visual encoding functions, For text semantic encoding functions, cos(·) is Cosine similarity calculation function The candidate visual frames in this set, Q For the input question text, For the ASR The set of candidate visual frames within the corresponding time interval of the segment. For the ASR The fragment corresponds to the visual frame in the candidate visual frame set that is most relevant to the question text.

[0059] Specifically, a second screening using visual-text similarity ensures that the selected visual frames are directly related to the semantics of the problem, forming an accurate set of supplementary visual frames to compensate for the limitations of basic uniform frame extraction.

[0060] Among them, the visual encoding function The candidate visual frame v is transformed into a visual feature vector, enabling cross-modal similarity calculation between visual information and text semantic vectors; x The function is used to select from the set of candidate visual frames. Filter out text related to the question Q The visual frame with the highest semantic similarity Ensure that each relevant ASR Within the time interval corresponding to the segment, only the visual frames with the highest evidentiary value for the answer are retained. This ensures the accuracy of the visual supplementary frame set while avoiding redundant visual frames that would increase the model's inference burden.

[0061] In one possible embodiment, the expression for the visual supplementary frame set in S2 is:

[0062] In the formula, For the visual supplementary frame set, For the ASRThe fragment corresponds to the visual frame in the candidate visual frame set that is most relevant to the question text. K For the ASR The number of segments.

[0063] Specifically, the expression will assign each relevant ASR The optimal visual frame corresponding to the segment Integrate into a visual supplementary frame set The main purpose is to achieve "visual keyframe aggregation guided by audio semantics". Each frame in the dataset is directly related to the semantics of the question and covers the main visual information of different relevant audio time intervals. Merging it with the base frame set can significantly improve the visual information quality of the input model and make up for the shortcomings of traditional uniform frame extraction in missing key frames in long videos.

[0064] In one possible embodiment, the expression for the base frame set obtained by uniform frame extraction in S3 is:

[0065] In the formula, For the basic frame set, For the first The aforementioned basic visual frames. For the first The timestamps corresponding to the basic visual frames. T The total number of frames in the uniform frame extraction, and T=32 ; The final visual input set is:

[0066] In the formula, For the final visual input set, This is the set of supplementary visual frames; The visual frame and the ASR The matching criteria for the fragment are: ,satisfy

[0067] In the formula, In order to be with the first The speech text matched by the visual frame. The spoken text content. For the first The timestamp of each of the visual frames. The start time corresponding to the spoken text. The end time corresponding to the spoken text.

[0068] Specifically, the base frame set obtained by uniform frame extraction ensures the coverage of global visual information in the video. After being merged with the visual supplementary frame set, it takes into account both the overall picture and highlights the key points. By matching timestamps, each visual frame is bound to corresponding audio text, so that each visual frame carries "speech semantics at the same moment". This allows the model to simultaneously acquire multimodal information of "image and corresponding audio" during inference, improving the completeness of semantic understanding. Serialization processing is to adapt to the input format of existing video language models without modifying the model's backbone structure, achieving plug-and-play characteristics.

[0069] Among them, the base frame set obtained by uniform frame extraction Used to cover global visual information in the video, avoiding the loss of global context due to supplementary frames that rely solely on audio guidance; T=32 The value of is determined with reference to the input frame count limit of mainstream video language models, balancing model processing efficiency and global visual coverage; each basic frame Bind timestamp For the subsequent and ASR Time alignment of fragments provides the necessary time signature. This is achieved through set merging operations. It incorporates both uniformly sampled global visual information and key visual supplementary frames guided by audio, achieving a visual input construction logic of "global coverage and emphasis on key points." This combination ensures that the model can capture the complete context of the video while focusing on the main visual evidence relevant to the question, providing a comprehensive and accurate visual foundation for cross-modal collaborative reasoning. The matching condition is the main rule for achieving "visual-to-audio" alignment, using the timestamps of the visual frames. and ASR The time interval of the segment [ , Perform interval matching to precisely bind the speech and text at the same moment to each visual frame. This strong temporal binding solves the problem in existing technologies where audio text is only used as global context and is disconnected from the semantics of visual frames. It enables the model to synchronously acquire the corresponding speech semantics when processing each visual frame, thereby improving the collaborative understanding of cross-modal information.

[0070] In one possible embodiment, the multimodal input representation in S4 and the model output obtained by inputting the question text into the video language model are as follows:

[0071] In the formula, The predicted answer to the question text. For the video language model, This is the final multimodal input representation. Q is The text of the question entered.

[0072] Specifically, the bidirectional aligned multimodal input contains complete visual information (base frame and supplementary frame) and realizes temporal and semantic collaboration between audio and video, solving the problem of separation between "seeing" and "hearing" in traditional input. The video language model performs reasoning based on the integrated multimodal semantics, which can make full use of the complementary information of visual evidence and audio semantics, significantly reducing the probability of reasoning failure caused by the omission of key information in long videos, and is especially suitable for complex scenarios such as long videos and dialogue-intensive content.

[0073] Among them, the serialized multimodal input representation Temporal and semantic alignment of the visual frames and corresponding audio text has been achieved, and this has been integrated with the question text. Q Input video language model This enables the model to utilize both visual evidence and aligned audio semantics during the reasoning process, fully leveraging the advantages of cross-modal collaboration. The output method follows the input-output specifications of existing video language models, requiring no modification to the model's backbone structure, achieving plug-and-play functionality. Furthermore, through bidirectional aligned input optimization, it directly improves the model's question-answering accuracy for long videos and complex semantic scenarios.

[0074] The application scenarios of this invention are as follows: Online education long video Q&A: Deployed in long video teaching scenarios such as vocational skills training and academic lectures, it uses a two-way audio-visual alignment mechanism to quickly locate the corresponding voice explanation and operation screen of knowledge points, providing students with accurate knowledge Q&A, and is compatible with the video Q&A model of existing education platforms.

[0075] Intelligent retrieval of film and television content: Deployed in long video storage scenarios such as film and television platforms and content libraries, it uses the two-way alignment function of "audio semantic positioning of screen + screen playback of audio" to provide users with accurate retrieval services related to lines and plots, and is compatible with existing film and television content Q&A systems.

[0076] Intelligent Q&A for Meeting Recording: Deployed in remote office and enterprise meeting recording scenarios, it utilizes frame-level ASR injection and audio-guided visual retrieval to provide professionals with key information query services such as decision-making and task allocation, and is compatible with the Q&A modules of existing meeting minutes tools.

[0077] Product after-sales video consultation: Deployed in after-sales scenarios for products such as home appliances and industrial equipment, it uses a two-way alignment mechanism to link operation demonstration screens with synchronous voice explanations, providing users with precise guidance on troubleshooting and function usage, and is compatible with the enterprise's existing after-sales video Q&A system.

[0078] Documentary / Interview Content Retrieval: Deployed on documentary libraries and interview program storage platforms, it utilizes audio semantic guidance and timestamp alignment to provide users with accurate queries of opinion- and event-related content, and is compatible with existing long-form video retrieval systems.

[0079] Autonomous driving video analysis: Deployed in autonomous vehicle driving recording scenarios, it utilizes audio and video time and semantic alignment to provide collaborative reasoning basis of "video + synchronized audio" for behavior review, and is compatible with existing in-vehicle video analysis models.

[0080] Medical training video Q&A: Deployed in medical education scenarios such as surgical teaching and instrument operation training, it uses a two-way alignment mechanism to locate the corresponding voice explanations and images for key operation points, providing medical staff with precise training Q&A, and is compatible with the existing Q&A model of the medical education platform.

[0081] Interactive features in the long video section of short video platforms: Deployed in the long video section of short video platforms, utilizing a plug-and-play two-way alignment enhancement solution, this feature provides users with a quick Q&A service for key information and is compatible with [other platforms / initiatives]. Qwen2.5-VL-7B Existing visual language models, etc.

[0082] It should be noted that the method of this embodiment can also be applied to distributed scenarios, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the long video question-answering enhancement processing method based on bidirectional audio-visual alignment.

[0083] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] Example 2

[0085] See Figure 4 This invention also provides a long video question-answering enhancement processing system based on bidirectional audio-visual alignment, which employs the aforementioned long video question-answering enhancement processing method based on bidirectional audio-visual alignment and includes the following modules: Audio processing and ASR Module 100 is used to acquire the input video and its corresponding audio stream, segment the audio using a sliding window approach, perform speech recognition on each audio segment, and integrate the transcription results of overlapping audio segments using a language model to generate... ASR Fragment collection; Audio to Visual Alignment A2VModule 200 is used to calculate the relationship between the input question text and the given question text. ASR The semantic similarity of each speech text in the fragment set is used to filter out ASR fragments relevant to the question. Among the filtered fragments... ASR Within the time interval corresponding to the segment, the video is intensively frame-by-frame extracted. Then, by calculating the visual-text similarity between the candidate frames and the question text, visual frames related to the question are selected to form a set of visual supplementary frames. Visual to audio alignment V2A Module 300 is used to perform uniform frame extraction on the input video to obtain a base frame set, merge the base frame set with the visual supplementary frame set to obtain a final visual input set, and, based on the timestamp of each visual frame in the final visual input set, perform... ASR The speech text within the corresponding time interval is matched from the fragment set. The speech text is used as a local audio context aligned with the visual frame time to form a multimodal input unit. The multimodal input unit is serialized into a unified multimodal input representation accepted by the video language model according to a predetermined format. The video question answering reasoning module 400 is used to input the multimodal input representation and question text into the video language model and output the predicted answer to the question.

[0086] It should be noted that the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here.

[0087] Example 3

[0088] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code for a long video question-answering enhancement processing method based on bidirectional audio-visual alignment. The program code includes instructions for executing the long video question-answering enhancement processing method based on bidirectional audio-visual alignment as described in Embodiment 1 or any possible implementation thereof.

[0089] Computer-readable storage media can be any usable medium that a computer can access, or a data storage device such as a server or data center that integrates one or more usable media. The usable medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g.,... DVD ), or semiconductor media (such as solid-state drives (SSDs) SolidStateDisk , SSD ))wait.

[0090] Example 4

[0091] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor; The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can call the program instructions to execute the long video question-answering enhancement processing method based on bidirectional audio-visual alignment of Embodiment 1 or any possible implementation thereof.

[0092] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0093] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another; for example, the computer instructions can be transmitted from a website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, digital subscriber line) communication. DSL It can transmit data to another website, computer, server, or data center via either a wireless (e.g., infrared, wireless, microwave) method or a wireless method.

[0094] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0095] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for enhancing question-answering in long videos based on bidirectional audio-visual alignment, characterized in that, Includes the following steps: S1, Audio Processing and ASR Construction: Obtain the input video and corresponding audio stream, segment the audio using a sliding window approach, perform speech recognition on each audio segment, and integrate the transcription results of overlapping audio segments using a language model to generate... ASR Fragment collection; S2, Audio-to-Visual Alignment A2V For the input question text, calculate the relationship between the question text and the input question text. ASR The semantic similarity of each audio text in the fragment set is used to filter out those relevant to the problem. ASR Fragments, in each of the selected... ASR Within the time interval corresponding to the segment, the video is intensively frame-by-frame extracted. Then, by calculating the visual-text similarity between the candidate frames and the question text, visual frames related to the question are selected to form a set of visual supplementary frames. S3, Visual-to-Audio Alignment V2A The input video is subjected to uniform frame extraction to obtain a base frame set. The base frame set is then merged with the visual supplementary frame set to obtain the final visual input set. Based on the timestamps of each visual frame in the final visual input set, the... ASR The speech text within the corresponding time interval is matched from the fragment set. The speech text is used as a local audio context aligned with the visual frame time to form a multimodal input unit. The multimodal input unit is serialized into a unified multimodal input representation accepted by the video language model according to a predetermined format. S4. Video Question Answering Reasoning: Input the multimodal input representation and question text into the video language model, and output the predicted answer to the question.

2. The method according to claim 1, characterized in that, The time interval of the audio segment divided by the sliding window in S1 is: ,and Δ <W In the formula, The aforementioned set of audio segments, The audio segment, W For window length, Δ The window step size; In S1, speech recognition is performed on each audio segment to obtain the initial transcribed text: In the formula, For the initial transcribed text, For the speech recognition; The generated in S1 ASR The expression for the fragment set is: In the formula, For the purposes of this statement ASR Fragment collection, This is voice-to-text content. The start time corresponding to the spoken text. This is the end time corresponding to the spoken text. N For the ASR Total number of fragments The aforementioned voice text; The ASR The fragment set includes the speech text content and the corresponding start time and end time of the speech text.

3. The method according to claim 1, characterized in that, The problem text mentioned in S2 and the... ASR The semantic similarity of audio-visual text segments is calculated as follows: In the formula, The semantic similarity is mentioned above. For text semantic encoding functions, cos(·) is Cosine similarity calculation function, This refers to the content of the spoken text. Q The input is the question text.

4. The method according to claim 1, characterized in that, S2 filters out the relevant questions. ASR The number of segments is K , K The value of is 8, and the set expression is: In the formula, For K of the aforementioned ASRs Fragment collection, The semantic similarity is mentioned above. for K ASRs Voice text Excerpt.

5. The method according to claim 1, characterized in that, The expression for the candidate visual frame set obtained by dense frame extraction in S2 is: In the formula, For the ASR The set of candidate visual frames within the corresponding time interval of the segment. The candidate visual frames in this set, The aforementioned voice text. M This represents the total number of densely drawn frames within that time interval.

6. The method according to claim 1, characterized in that, The method for calculating visual-text similarity as described in S2 is as follows: In the formula, For visual encoding functions, For text semantic encoding functions, cos(·) is Cosine similarity calculation function The candidate visual frames in this set, Q For the input question text, For the ASR The set of candidate visual frames within the corresponding time interval of the segment. For the ASR The fragment corresponds to the visual frame in the candidate visual frame set that is most relevant to the question text.

7. The method according to claim 1, characterized in that, The expression for the visual supplementary frame set mentioned in S2 is: In the formula, For the visual supplementary frame set, For the ASR The fragment corresponds to the visual frame in the candidate visual frame set that is most relevant to the question text. K For the ASR The number of segments.

8. The method according to claim 1, characterized in that, The expression for the base frame set obtained by uniform frame extraction in S3 is: In the formula, For the basic frame set, For the first The aforementioned basic visual frames. For the first The timestamps corresponding to the basic visual frames. T The total number of frames in the uniform frame extraction, and T=32 ; The final visual input set is: In the formula, For the final visual input set, This is the set of supplementary visual frames; The visual frame and the ASR The matching criteria for the fragment are: ,satisfy In the formula, In order to be with the first The speech text matched by the visual frame. The spoken text content. For the first The timestamp of each of the visual frames. The start time corresponding to the spoken text. The end time corresponding to the spoken text.

9. The method according to claim 1, characterized in that, The multimodal input representation described in S4 and the model output method obtained by inputting the question text into the video language model are as follows: In the formula, The predicted answer to the question text. For the video language model, This is the final multimodal input representation. Q is The text of the question entered.

10. A long video question-answering enhancement processing system based on bidirectional audio-visual alignment, employing the long video question-answering enhancement processing method based on bidirectional audio-visual alignment as described in any one of claims 1-9, characterized in that, Includes the following steps: Audio processing and ASR The module is used to acquire the input video and its corresponding audio stream, segment the audio using a sliding window approach, perform speech recognition on each audio segment, and integrate the transcription results of overlapping audio segments using a language model to generate a... ASR Fragment collection; Audio to Visual Alignment A2V This module is used to calculate the relationship between the input question text and the given question text. ASR The semantic similarity of each audio text in the fragment set is used to filter out those relevant to the problem. ASR Fragments, in each of the selected... ASR Within the time interval corresponding to the segment, the video is intensively frame-by-frame extracted. Then, by calculating the visual-text similarity between the candidate frames and the question text, visual frames related to the question are selected to form a set of visual supplementary frames. Visual to audio alignment V2A The module is used to perform uniform frame extraction on the input video to obtain a base frame set, merge the base frame set with the visual supplementary frame set to obtain a final visual input set, and, based on the timestamp of each visual frame in the final visual input set, perform... ASR The speech text within the corresponding time interval is matched from the fragment set. The speech text is used as a local audio context aligned with the visual frame time to form a multimodal input unit. The multimodal input unit is serialized into a unified multimodal input representation accepted by the video language model according to a predetermined format. The video question-answering reasoning module is used to input the multimodal input representation and question text into the video language model and output the predicted answer to the question.

Images