Reference audio and video segmentation method based on multi-agent identification, reasoning and reflection

Abstract

A reference audio and video segmentation method based on multi-agent identification, reasoning and reflection belongs to the technical field of reference audio and video segmentation in the field of computer vision and multi-modal understanding. The present application is aimed at the problem that the existing reference audio and video segmentation method cannot explicitly identify the modal contribution and task difficulty according to the reference expression, which affects the segmentation result. It includes analyzing the reference expression by using the consensus multi-modal identification mechanism to obtain the task difficulty, the dominant mode and its reason; calling the corresponding dominant mode agent and auxiliary mode agent to complete the reasoning of the initial target object; inputting the obtained initial target object as a text prompt into the segmentation agent to generate an initial segmentation mask; finally, the consistency between the intermediate segmentation result and the original reference expression is checked to generate a corrected segmentation mask. The present application can realize reliable reasoning and accurate segmentation of the reference target in a complex dynamic audio and video scene.

Description

Technical Field

[0001] This invention relates to a reference audio and video segmentation method based on multi-agent recognition, reasoning, and reflection, belonging to the field of reference audio and video segmentation technology in the fields of computer vision and multimodal understanding. Background Technology

[0002] With the continuous development of artificial intelligence, multimodal reasoning and understanding have become core research topics in the interdisciplinary field of computer vision and embodied intelligence. Inspired by humans' perception of the physical world through the collaboration of multiple senses such as sight, hearing, and touch, how to enable machine systems to achieve deep reasoning and pixel-level segmentation of audio and video information in complex dynamic environments has attracted widespread attention from academia and industry. Among them, Reference Audio-Visual Segmentation (Ref-AVS), as a highly challenging multimodal task, aims to accurately segment the target object referred to by a given natural language reference expression in a video sequence containing audio. This technology has shown great application potential and commercial value in fields such as automated post-production of film, intelligent video editing, assisted driving in complex scenarios, and human-computer interaction systems.

[0003] In complex dynamic audio and video scenarios, achieving high-quality Ref-AVS tasks faces multiple challenges, including effective integration of multimodal information, deep understanding of natural language references, and precise pixel-level localization. Because audio signals and visual images in video are highly coupled in both time and space, and reference expressions often involve highly abstract semantic constraints (such as "the object whose sound is shorter than a piano"), existing technologies still have significant limitations in processing this highly correlated information with temporal dynamic characteristics.

[0004] Existing Ref-AVS techniques can be broadly categorized into two types. The first type focuses on implicit fusion of multimodal features. These methods typically utilize predefined feature extraction networks to acquire audio, video, and text features separately. Then, a multimodal fusion module (such as a Transformer architecture or cross-attention mechanism) maps these features across different dimensions into a unified semantic space, using this as a learned embedding representation or cue information to guide the segmentation decoder in generating a binary mask for the target. However, this feature fusion-based approach suffers from a fundamental flaw: its implicit processing of multimodal inputs completely ignores the explicit identification of modal contributions and task difficulty within the reference expressions. In practical applications, different reference expressions exhibit significantly different degrees of dependence on audio and visual cues. Existing methods often employ a uniform implicit fusion strategy, failing to explicitly analyze the cognitive difficulty of the task itself and the weight contributions of each modality before inference execution.

[0005] The second type of approach attempts to introduce multimodal large language models (MLLMs) for complex target reasoning. This type of approach typically requires instruction-tuning to improve the model's understanding of the reference audio / video segmentation task, enabling it to convert complex audio / video input into textual descriptions of the target object, and then combining it with external segmentation models (such as SAM and SAM2) to complete the task. While introducing large models enhances the system's logical reasoning depth, research has found two significant drawbacks: First, the contradiction between computational overhead and resource consumption. MLLMs typically have billions or even tens of billions of parameters, making instruction-tuning excessively demanding on GPU memory and computing resources. Second, the high sensitivity to the tuning data. The performance of this type of method depends heavily on the size and quality of the instruction-tuning dataset.

[0006] The more critical technical challenge lies in the fact that existing Ref-AVS schemes generally employ a single-stage prediction paradigm, severely lacking posterior reflection and self-correction mechanisms. When humans perform similar complex cognitive tasks, they typically follow a closed-loop logic of "instruction analysis - initial attempt - result verification - bias correction." In contrast, existing technologies terminate the process after generating the initial segmentation mask. Due to the lack of closed-loop verification of the consistency between intermediate segmentation results and the original reference instructions, the model cannot perceive whether it has incurred biases during the inference phase. For example, if the initial stage mistakenly identifies "hair dryer" as "car," even if subsequent segmentation algorithms produce pixel-perfect masks, the result is completely semantically invalid.

[0007] In summary, there is an urgent need in this field for a reference audio and video segmentation method that does not require expensive fine-tuning, does not rely on domain-specific big data, and saves computational resources. Summary of the Invention

[0008] To address the problem that existing reference audio and video segmentation methods cannot explicitly identify modal contribution and task difficulty based on reference expressions, thus affecting the segmentation results, this invention provides a reference audio and video segmentation method based on multi-agent recognition, reasoning, and reflection.

[0009] This invention provides a reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection, comprising:

[0010] Consensus Multimodal Recognition: Three different multimodal cue analysis agents are used to output task difficulty, dominant modality, and reasoning based on a reference expression. These agents then interact with each other, and a final decision-making agent obtains the final task difficulty, dominant modality, and reasoning. Task difficulty is categorized as low, medium, and high. Low difficulty corresponds to tasks where target recognition relies on a single dominant modality; medium difficulty corresponds to tasks where target recognition relies on one dominant modality and another auxiliary modality; and high difficulty corresponds to tasks where target recognition relies on two dominant modalities simultaneously.

[0011] Collaborative goal reasoning: Based on the final task difficulty and dominant modality obtained in the consensus multimodal recognition stage, the visual agent, audio agent, and audio-visual agent are adaptively organized to collaboratively process video frames, synchronized audio, and reference expressions to determine the initial recognition target and reasoning.

[0012] Reflective learning segmentation: A segmentation agent generates an initial mask for video frames based on the initial recognition target obtained in the collaborative goal inference stage, obtaining an initial segmentation result. Then, a checking agent judges the consistency between the initial segmentation result and the reference expression based on the video frames with the initial mask, synchronized audio, and reference expression. If they are inconsistent, a corrected recognition target is output, and the segmentation agent generates a corrected mask for the video frames, obtaining a corrected segmentation result. The checking agent then judges the consistency between the corrected segmentation result and the reference expression based on the video frames with the corrected mask, synchronized audio, reference expression, and the corrected recognition target obtained in the previous iteration, until the checking agent judges that they are consistent or the maximum number of iterations is reached, obtaining the final segmentation result.

[0013] According to the reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection of the present invention, in the consensus multimodal recognition stage, the method for peer interaction among the three multimodal cue analysis agents is as follows:

[0014] Each multimodal cue analysis agent reads the task difficulty, dominant mode, and rationale output by the other two multimodal cue analysis agents, re-analyzes the reference expression, and obtains the revised task difficulty, dominant mode, and rationale.

[0015] According to the reference audio-visual segmentation method based on multi-agent recognition, reasoning and reflection of the present invention, the final decision agent obtains the final task difficulty, dominant mode and reason based on the modified task difficulty, dominant mode and reason output by the three multimodal cues analysis agents respectively.

[0016] According to the reference audio-visual segmentation method based on multi-agent recognition, reasoning and reflection of the present invention, in the collaborative target reasoning stage, for low-difficulty tasks, if the dominant modality is the visual modality, the visual agent receives video frames and reference expressions, and outputs the initial recognition target and reasoning.

[0017] If the dominant mode is the audio mode, the audio agent receives the synchronized audio and reference expression, and outputs the initial recognition target and reason.

[0018] According to the reference audio-video segmentation method based on multi-agent recognition, reasoning and reflection of the present invention, in the collaborative target reasoning stage, for a medium-difficulty task, if the dominant modality is the visual modality and the auxiliary modality is the audio modality, the audio agent first receives the synchronized audio and reference expression, and outputs a candidate target list and reasons; then the visual agent outputs the initial recognition target and reasons based on the candidate target list and reasons, combined with the video frame and reference expression.

[0019] If the dominant modality is the audio modality and the auxiliary modality is the visual modality, then the visual agent first receives the video frames and reference expressions, and outputs a list of candidate targets and reasons; then the audio agent, based on the list of candidate targets and reasons, combined with the synchronized audio and reference expressions, outputs the initial recognition target and reasons.

[0020] According to the reference audio-video segmentation method based on multi-agent recognition, reasoning, and reflection of the present invention, in the collaborative target reasoning stage, for high-difficulty tasks, the audio agent first receives synchronized audio and reference expressions, and outputs a candidate target list and reasons based on the audio; simultaneously, the visual agent receives video frames and reference expressions, and outputs a candidate target list and reasons based on the video; then, the audio-video agent, based on the candidate target list and reasons based on the audio and the candidate target list and reasons based on the video, combined with the synchronized audio, video frames, and reference expressions, outputs the initial identified target and reasons.

[0021] According to the reference audio-visual segmentation method based on multi-agent recognition, reasoning and reflection of the present invention, in the reflective learning segmentation stage, the segmentation agent uses the initial recognition target as a text prompt to generate an initial mask for the video frame.

[0022] According to the reference audio-visual segmentation method based on multi-agent recognition, reasoning and reflection of the present invention, in the consensus multimodal recognition stage, the dominant modality of low-difficulty tasks is the audio modality or the visual modality.

[0023] For medium-difficulty tasks, the dominant modality is either the audio or visual modality, and the auxiliary modality is either the visual or audio modality.

[0024] The dominant modalities for high-difficulty tasks are audio and visual modalities.

[0025] The beneficial effects of this invention are as follows: The method of this invention is applicable to complex dynamic audio and video scenarios, under conditions of limited large-scale instruction fine-tuning data and computational resources. It can explicitly identify task difficulty and modal contribution, possess adaptive multi-agent collaborative reasoning capabilities, and achieve self-correction through a reflective mechanism, thereby achieving reference audio and video segmentation. This method does not require expensive fine-tuning, does not rely on domain-specific big data, and saves computational resources. By simulating advanced human cognitive processes, it solves the problems of insufficient multimodal semantic alignment accuracy and unreliable reasoning results in complex dynamic scenarios, fundamentally achieving higher quality and more robust audio and video target segmentation.

[0026] This invention addresses the shortcomings of existing Reference Audio-Visual Segmentation (Ref-AVS) methods in complex dynamic audio-visual scenarios, including insufficient recognition of reference expression difficulty, inadequate modeling of modality dominance relationships, strong dependence of target inference on high-quality instruction fine-tuning data, and lack of a reflective verification mechanism for intermediate segmentation results. It comprises three parts: Consensus Multimodal Recognition (CMR), Collaborative Object Reasoning (COR), and Reflective Learning Segmentation (RLS). By explicitly identifying the difficulty level and dominant modality of the reference expression before task execution, it adaptively organizes the collaborative reasoning process between different modal agents. After the initial segmentation, a Check agent is introduced to verify intermediate results and correct object prompts, thereby improving the deep understanding of multimodal cues, the reliability of target inference, and the accuracy of mask prediction in complex dynamic audio-visual scenarios. Attached Figure Description

[0027] Figure 1 This is an overall block diagram of the reference audio and video segmentation method based on multi-agent recognition, reasoning, and reflection described in this invention;

[0028] Figure 2 This is a visual comparison of the method of the present invention and existing methods on different examples;

[0029] Figure 3 This is a distribution chart showing the percentage of recognition difficulty of the reference expression using the method of this invention on the Ref-AVSBench test set;

[0030] Figure 4 This is a distribution diagram of the dominant modality proportion of the reference expression identified by the method of the present invention on the Ref-AVSBench test set. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Specific Implementation Method 1: Combination Figure 1 As shown, this invention provides a reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection, including:

[0033] Consensus Multimodal Recognition: Three different multimodal cue analysis agents are used to output task difficulty, dominant modality, and reasoning based on a reference expression. These agents then interact with each other, and a final decision-making agent obtains the final task difficulty, dominant modality, and reasoning. Task difficulty is categorized as low, medium, and high. Low difficulty corresponds to tasks where target recognition relies on a single dominant modality; medium difficulty corresponds to tasks where target recognition relies on one dominant modality and another auxiliary modality; and high difficulty corresponds to tasks where target recognition relies on two dominant modalities simultaneously.

[0034] Collaborative goal reasoning: Based on the final task difficulty and dominant modality obtained in the consensus multimodal recognition stage, the visual agent, audio agent, and audio-visual agent are adaptively organized to collaboratively process video frames, synchronized audio, and reference expressions to determine the initial recognition target and reasoning.

[0035] Reflective learning segmentation: A segmentation agent generates an initial mask for video frames based on the initial recognition target obtained in the collaborative goal inference stage, obtaining an initial segmentation result. Then, a checking agent judges the consistency between the initial segmentation result and the reference expression based on the video frames with the initial mask, synchronized audio, and reference expression. If they are inconsistent, a corrected recognition target is output, and the segmentation agent generates a corrected mask for the video frames, obtaining a corrected segmentation result. The checking agent then judges the consistency between the corrected segmentation result and the reference expression based on the video frames with the corrected mask, synchronized audio, reference expression, and the corrected recognition target obtained in the previous iteration, until the checking agent judges that they are consistent or the maximum number of iterations is reached, obtaining the final segmentation result.

[0036] In the consensus multimodal recognition stage, the dominant modality for low-difficulty tasks is either the audio modality or the visual modality;

[0037] For medium-difficulty tasks, the dominant modality is either the audio or visual modality, and the auxiliary modality is either the visual or audio modality.

[0038] The dominant modalities for high-difficulty tasks are audio and visual modalities.

[0039] This implementation focuses on segmenting video sequences with synchronized audio. The input includes a video frame sequence, the corresponding synchronized audio stream, and a natural language reference expression describing the target to be segmented. Compared to traditional visual segmentation tasks, Ref-AVS requires not only recognizing the visual appearance of the target but also understanding multimodal information such as sound events, action relationships, spatial relationships, and comparative relationships. Therefore, the reference expression may rely solely on a single visual cue to determine the target, or it may need to combine information such as sound duration, speaker, spatial location, and the relationships between characters' actions for joint determination.

[0040] After preparing the input data, this implementation first inputs the reference expression into the consensus multimodal recognition mechanism. Multiple multimodal cue analysis agents (MCA agents) independently determine the task difficulty, dominant modality, and reason corresponding to the expression, and form a unified analysis result through independent thinking, peer interaction, and final decision. Subsequently, based on the identified difficulty level and dominant modality, the corresponding dominant-modality agent and auxiliary-modality agent are invoked in the collaborative goal reasoning mechanism to reason about the initially referred object referred to by the reference expression, outputting the initial target object and its reasoning basis. Finally, the initial target object is used as a text prompt input to the segmentation agent to generate an initial segmentation mask. The segmentation agent then checks the current segmentation result using video, audio, and a reference expression. If the current highlighted target is inconsistent with the reference expression, the target object prompt is corrected, and the revised Referred Object Prompt is re-inputted into the segmentation agent to obtain a more accurate segmentation result.

[0041] To address the lack of explicit identification of reference expression difficulty and modal dominance relationships in existing technologies, this implementation proposes a consensus-based multimodal recognition mechanism. This mechanism draws on the multi-round consensus concept of the Delphi Method, constructing multiple multimodal cue analysis agents to analyze the same reference expression. Specifically, a modality-dominant difficulty rule is first established: when target recognition relies solely on a single modality from the audio (or visual) modality, it is classified as low difficulty; when one modality is dominant and the other is auxiliary, it is classified as medium difficulty; and when both the audio and visual modalities are crucial, it is classified as high difficulty. This mechanism allows for the clarification of expression complexity and modal contribution relationships before task execution, providing a basis for subsequent inference path selection.

[0042] To address the issue that a unified reasoning strategy struggles to adapt to expressions of varying difficulty, this implementation proposes a collaborative goal-oriented reasoning mechanism. This mechanism adaptively selects different goal-oriented reasoning paths based on the difficulty level and dominant modality output by the consensus multimodal recognition mechanism: in low-difficulty cases, the dominant modality agent directly completes the goal-oriented reasoning; in medium-difficulty cases, the auxiliary modality agent first generates a candidate object list and reasons, and then the dominant modality agent combines auxiliary information to complete the reasoning; in high-difficulty cases, the audio-assisted agent and the visual-assisted agent respectively generate candidate objects and reasons, and then the audio-visual dominant agent fuses audio streams, video frames, reference expressions, and dual auxiliary information to jointly complete the initial goal-oriented reasoning. Through this adaptive collaborative reasoning strategy, this implementation effectively utilizes complementary information between different modalities, reducing the risk of misjudgment caused by insufficient perception of a single modality.

[0043] To address the lack of posterior error correction capabilities in existing single-stage segmentation paradigms, this implementation proposes a reflective learning segmentation mechanism. This mechanism introduces a checking agent after the initial segmentation to verify intermediate segmentation results and correct target object text prompts when necessary. Specifically, the segmentation agent generates an initial mask based on the initial target object; the checking agent receives video with the currently highlighted target, synchronized audio, a reference expression, and the target object prompts from the previous round, and judges whether the current segmentation result matches the reference expression. If the judgment result is a match, the current result is retained; if the judgment result is a mismatch, a corrected target object prompt is generated and re-segmentation is performed. The corrected prompt only describes the target object itself that matches the reference expression. Through this reflective learning segmentation mechanism, a closed loop of result verification and object prompt correction is established, effectively improving the semantic consistency, accuracy, and robustness of the final segmentation result.

[0044] This implementation method achieves a deep understanding of the reference expression, reliable reasoning of the referred object, and adaptive correction of the segmentation results through three stages: consensus multimodal recognition, collaborative target reasoning, and reflective learning segmentation. It is a novel reference audio and video segmentation method applicable to real and complex audio and video scenarios.

[0045] This implementation is based on the Reference Audio-Visual Segmentation (Ref-AVS) task and employs a Multi-Agent Recognition, Reasoning, and Reflection (MAR3) framework. At the input end, it receives a video sequence with synchronized audio, the corresponding audio stream, and a natural language reference expression. At the output end, it generates a pixel-level segmentation mask for the target object referred to by the reference expression. Unlike existing methods, this implementation does not directly feed the multimodal input into a unified model for implicit fusion or single-stage segmentation. Instead, it first explicitly identifies the difficulty level and dominant modality of the reference expression, then performs adaptive collaborative target reasoning based on the identification results, and finally uses a reflective learning mechanism to perform consistency verification and target text prompt correction on the initial segmentation results, thereby achieving high-quality reference audio-visual segmentation. Figure 1The framework diagram clearly shows that the method of this invention consists of three parts: Consensus Multimodal Recognition (CMR), Collaborative Object Reasoning (COR), and Reflective Learning Segmentation (RLS).

[0046] Combination Figure 1 The overall technical process of this implementation can be summarized in the following steps. First, prepare the audio and video samples to be processed and their reference expressions according to the actual application requirements, providing an input basis for subsequent recognition, reasoning, and segmentation. Then, analyze the reference expressions using a consensus multimodal recognition mechanism to explicitly obtain the task difficulty, dominant modality, and its cause. Next, based on the identified difficulty level and dominant modality, call the corresponding dominant modality agent and auxiliary modality agent to complete the reasoning of the initial target object. Based on this, input the reasoned initial target object as a text prompt into the segmentation agent to generate an initial segmentation mask. Finally, check the consistency between the intermediate segmentation results and the original reference expression using a checking agent; if a mismatch is found, the checking agent generates a corrected target object prompt, re-inputs it into the segmentation agent, and iteratively generates a corrected segmentation mask. Through the above closed-loop process, reliable reasoning and accurate segmentation of the reference target can be achieved in complex dynamic audio and video scenarios.

[0047] Furthermore, in the consensus multimodal recognition stage, the three methods for peer interaction among agents in multimodal cue analysis are as follows:

[0048] Each multimodal cue analysis agent reads the task difficulty, dominant mode, and rationale output by the other two multimodal cue analysis agents, re-analyzes the reference expression, and obtains the revised task difficulty, dominant mode, and rationale.

[0049] The final decision-making agent obtains the final task difficulty, dominant mode, and reason based on the modified task difficulty, dominant mode, and reason output by the three multimodal cues.

[0050] The consensus multimodal identification stage aims to explicitly identify the task difficulty and dominant modality corresponding to the reference expression, thereby alleviating the problem of existing methods ignoring expression complexity and modal contribution relationships during multimodal input processing. For example... Figure 1As shown, the input reference expression is first simultaneously fed into multiple MCA Agents for analysis. To improve the robustness of the recognition results, the multi-round consensus concept of the Delphi Method is adopted, dividing the recognition process into three stages: independent thinking, peer interaction, and final decision-making. Specifically, in the independent thinking stage, multiple MCA Agents receive the same reference expression and output the task difficulty, dominant modality, and reasoning respectively according to the preset modality-dominant difficulty rule.

[0051] During the peer interaction phase, each MCA Agent further reads the analysis results provided by other agents during the independent thinking phase, including difficulty, dominant modality, and rationale. Based on these results, it re-analyzes the original reference expression, thereby reducing the bias that may arise from a single agent's analysis. Then, in the final decision-making phase, the final decision-making agent synthesizes the discussion results generated by multiple MCA Agents during the peer interaction phase to form a unified recognition conclusion. Through this process, the difficulty level and dominant modality corresponding to the reference expression can be accurately obtained before task execution, providing a clear basis for the selection of subsequent collaborative goal reasoning paths.

[0052] In the collaborative target reasoning stage, for low-difficulty tasks, if the dominant modality is the visual modality, the visual agent receives video frames and reference expressions, and outputs the initial recognition target and reason.

[0053] If the dominant mode is the audio mode, the audio agent receives the synchronized audio and reference expression, and outputs the initial recognition target and reason.

[0054] In the collaborative target reasoning stage, for medium-difficulty tasks, if the dominant modality is the visual modality and the auxiliary modality is the audio modality, the audio agent first receives the synchronized audio and reference expression, and outputs a list of candidate targets and reasons; then the visual agent outputs the initial identified target and reasons based on the list of candidate targets and reasons, combined with video frames and reference expressions.

[0055] If the dominant modality is the audio modality and the auxiliary modality is the visual modality, then the visual agent first receives the video frames and reference expressions, and outputs a list of candidate targets and reasons; then the audio agent, based on the list of candidate targets and reasons, combined with the synchronized audio and reference expressions, outputs the initial recognition target and reasons.

[0056] In the collaborative target reasoning stage, for high-difficulty tasks, the audio agent first receives synchronized audio and a reference expression, and outputs a candidate target list and reasons based on the audio; at the same time, the visual agent receives video frames and a reference expression, and outputs a candidate target list and reasons based on the video; then, the audio-visual agent, based on the candidate target list and reasons based on the audio and the candidate target list and reasons based on the video, combined with the synchronized audio, video frames, and reference expressions, outputs the initial identified target and reasons.

[0057] The collaborative target reasoning phase aims to adaptively organize the collaboration between agents of different modalities based on the difficulty level and dominant modality given in the consensus multimodal recognition phase, thereby improving the reliability of initial target reasoning. Unlike a fixed unified reasoning strategy, this implementation divides the reasoning process into three categories—low difficulty, medium difficulty, and high difficulty—based on the complexity of the reference expression. In the low difficulty scenario, the corresponding visual agent or audio agent independently completes the target reasoning, outputting the initial target object and its rationale. In this case, since target judgment relies on only a single modality, there is no need to introduce additional auxiliary agents to participate in the reasoning. In the medium difficulty scenario, when the dominant modality is visual and the auxiliary modality is audio, the audio auxiliary agent first receives the audio stream and reference expression, generating a candidate object list and its rationale. Subsequently, this candidate object list and rationale are passed as auxiliary information to the visual dominant agent, which combines video frames, the reference expression, and the audio auxiliary information to reason about the initial target object. When the dominant modality is audio and the auxiliary modality is vision, a corresponding symmetrical process is executed. This design effectively utilizes supplementary cues provided by the auxiliary modality while maintaining the decision-making status of the dominant modality.

[0058] For high-difficulty scenarios, both audio and visual modalities are indispensable in target determination. Therefore, two auxiliary agents and one audio-visual dominant agent are needed to participate in the reasoning process. The audio-visual dominant agent simultaneously receives audio streams, video frames, reference expressions, and candidate information and reasons from the two auxiliary agents, jointly completing the initial target reasoning, thereby achieving multimodal joint decision-making. Through the collaborative reasoning design with low, medium, and high difficulty branches, this implementation can effectively utilize the complementary information between different modalities and reduce the risk of misjudgment caused by single-modal reasoning.

[0059] During the reflective learning segmentation phase, the segmentation agent uses the initial target as a text prompt to generate an initial mask for the video frame.

[0060] The reflexive learning segmentation phase aims to overcome the lack of posterior error correction capabilities in existing single-stage segmentation paradigms. It consists of two phases: Initial Segmentation and Iterative Segmentation. In the initial segmentation phase, the segmentation agent receives the initial target object output by the collaborative target inference module and inputs this object as a textual cue into the segmentation model to generate an initial mask for the video frame. Since this initial target object comes from the multi-agent inference of the previous phase, this cue provides the semantic basis for the initial segmentation. In the iterative segmentation phase, the checking agent receives the video with the currently highlighted target, synchronized audio, a reference expression, and the target object cue from the previous round, and judges whether the current segmentation result is consistent with the reference expression. The checking agent generates a corrected target object cue and re-inputs it into the segmentation agent to generate a new segmentation mask. Through the above checking and correction process, the reflexive learning segmentation module establishes a closed loop of result verification and object cue correction after initial inference and initial segmentation, thereby effectively correcting the deviation of the target inference in the previous phase.

[0061] Figure 1 In the consensus multimodal recognition stage, the reference expression is: the object the man on the right is playing. In the independent thinking stage, the three multimodal cue analysis agents output the following results: 1) Difficulty: High; Dominant modality: Audio + Vision; Reason: This task requires combining audio and video to determine the object...; 2) Difficulty: Medium; Dominant modality: Vision; Auxiliary modality: Audio; Reason: This expression relies more on vision than audio when locating the object...; 3) Difficulty: Low; Dominant modality: Vision; Reason: This task relies only on video to locate the object because...; The final decision agent outputs the following reason: This expression mainly requires visual location of the man and the object, while audio provides supplementary information for this action. In the collaborative target reasoning phase, the initial target output by the visual agent in the first frame is: double bass; the reason is: the finger indicates that it clearly points out...; the initial target output by the visual agent in the second frame is: double bass; the reason is: the object identified by the finger is...; the initial target output by the audio-visual agent in the third frame is: double bass; the reason is: there is a prominent double bass melody in the audio, and the finger indicates that it is pointing to the object that the man on the right is playing....

[0062] Specific Implementation: The first step is the preparation of input data and the implementation environment. Input data includes video frame sequences, corresponding audio streams, and natural language reference expressions describing the target objects. The Ref-AVSBench dataset can be selected as the experimental basis. This dataset contains 4000 audio videos with pixel-level annotations and reference expressions, covering 51 target categories. Of these, 2908 videos are used for training, 276 for validation, and 818 for testing. The test set is further divided into Seen and Unseen subsets. The Seen subset consists of 39 target categories that appeared during training, while the Unseen subset contains 13 target categories that did not appear during training. Through this dataset division, the segmentation capability of the present invention on seen and unseen categories can be comprehensively tested.

[0063] Regarding specific implementation parameters, the video sampling frame rate is set to 1 frame per second, and the audio sampling rate is set to 22050 Hz to balance temporal information preservation and computational efficiency. In the consensus multimodal recognition mechanism, the Multimodal Cues Analysis Agent (MCA Agent) is constructed using a heterogeneous large language model, with MCA Agent #1 using Qwen3-30B-A3B-Instruct, MCA Agent #2 using GLM4-32B, and MCA Agent #3 and the final decision-making agent using DeepSeek-R1-32B. In the collaborative goal reasoning mechanism, the Visual Agent, Audio Agent, and Audio-Visual Agent all use Qwen3-Omni-30B-A3B-Instruct. In the reflexive learning segmentation mechanism, the checking agent uses Qwen3-Omni-30B-A3B-Instruct, and the segmentation agent uses SAM3. The maximum number of reflexive iterations is set to 2.

[0064] Experimental Results: Unlike large-model segmentation schemes that rely on high-quality instruction fine-tuning data, this invention employs a training-free multi-agent workflow, decomposing the complex Ref-AVS task into three sub-tasks: "recognition—reasoning—reflection," thereby reducing the dependence on additional training resources. As shown in Table 1, experimental results on the Ref-AVSBench dataset demonstrate that this invention achieves a J&F of 69.2 on the Mix (Seen + Unseen) subset, a 3.4 percentage point improvement over the second-place TGS-Agent; a J&F of 62.2 on the Seen subset, showing a more significant improvement over the comparison methods; and a J&F of 76.2 on the Unseen subset, close to the performance of the comparison methods. These results indicate that this invention can still achieve strong reference audio and video segmentation capabilities without requiring additional training resources.

[0065] Table 1. Comparison with other methods on the Ref-AVSBench dataset.

[0066]

[0067] In Table 1, each subset has 。 " indicates that a higher value is better. Although the results marked with an asterisk (*) do not achieve optimal performance, the performance difference is still minimal."

[0068] Image visualization results further demonstrate the effectiveness of the invention: such as Figure 2 As shown in (a), the consensus multimodal recognition mechanism based on the Delphi method can effectively identify the difficulty level of the reference expression and the dominant modality among multimodal cues. Through this process, the collaborative target reasoning mechanism can adaptively integrate relevant auxiliary information from audio and visual modalities under the collaborative reasoning paradigm, thereby achieving more accurate and reliable reasoning about the initial target object; at the same time, the reflective learning segmentation mechanism can further output a more accurate target segmentation mask. Figure 2 As shown in (b), similar to the situation where TGS-Agent misidentifies "dog" as "the object with the longest vocalization duration," this invention may also infer the wrong target object in the initial stage. However, the proposed reflective learning mechanism can quickly check the intermediate segmentation results and generate a corrected target object prompt, "hair dryer," accordingly. This type of example further demonstrates that this invention has a significant advantage over baseline methods in achieving high-quality reference audio and video segmentation.

[0069] Figure 2As shown in (a), the TGS-Agent outputs the following result: The reference expression is: "Target emitting sound through singing and ukulele playing." The video shows a girl playing a red ukulele and singing. The audio includes the sound of the ukulele being played and the girl's singing. This reference expression involves both visual and auditory modalities, designating the ukulele as the sound source. Throughout the video, the ukulele is held by the girl and remains relatively still. Target: Ukulele.

[0070] The output of this invention is as follows: CMR: Difficulty: High; Dominant mode: Audio and video; Reason: Audio and video are both essential for identifying a target object that is simultaneously emitting singing and ukulele sounds, therefore the task is of high difficulty.

[0071] COR: Audio auxiliary information [girl, ukulele]; The audio contains a female vocal performance accompanied by ukulele playing; Visual auxiliary information: [girl, ukulele]; The reference expression describes a target object that produces sound through singing and ukulele playing. The girl in the image is actively playing the ukulele, and her mouth is slightly open, indicating that she may be singing; the ukulele is a sound-producing instrument; both, as described in the reference expression, directly participate in the sound production process. Initial referential target discrimination rationale: The visual representation shows the girl playing the ukulele, and her mouth movements indicate that she is singing, which matches the female vocal performance with ukulele accompaniment. Initial referential target: girl; RLS: Match: Yes.

[0072] Figure 2 As shown in (b), the TGS-Agent output is: The reference expression is: the object with the longest duration of sound. The video shows a puppy lying on a wooden floor. A person is holding a hairdryer and pointing it at the puppy. The audio contains a clearly identifiable whimpering or moaning sound from the puppy, and this sound persists throughout the video. The referent is modally related to the audio, specifically referring to the duration of the puppy's whimpering. The puppy remains largely still, with only slight movements, while the hairdryer is held and moved slightly. Keyframes 1, 5, and 10 all capture the puppy's continued presence and the position of the hairdryer. Referent: Dog.

[0073] The output of this invention is as follows: CMR: Difficulty: Medium; Dominant Modality: Audio; Auxiliary Modality: Visual; Rationale: This representation primarily relies on audio to measure the duration of sound, therefore audio is the dominant modality. However, video is also necessary as an auxiliary modality because audio alone cannot definitively and accurately map the sound to the correct object. Therefore, the difficulty of this task is medium, and audio is the dominant modality.

[0074] COR: Visual Auxiliary Information: [dog, hair dryer]; This reference expression specifies the object with the longest vocal duration. In the video, the hair dryer is the source of a continuous, loud buzzing sound, while the dog's sounds (such as growling or barking) are intermittent; Initial referent rationale: The dog is the only object in the audio that exists throughout the entire segment's duration, therefore it is the object with the longest vocal duration. Initial referent: dog; RLS: Match: No; Corrected referent: hair dryer.

[0075] The difficulty of the reference expression and the distribution of the dominant modes. Figure 3 The distribution of difficulty percentages for reference expressions on the Ref-AVSBench test set is shown. Statistical results indicate that... Figure 3 As shown in (a), the Mix(Seen+Unseen) subset has the highest proportion of complex expressions, reaching 66.8%, indicating that the test sample as a whole is dominated by complex expressions, verifying the necessity of identifying the difficulty of the reference expressions before task execution in this invention. Further comparison shows that, as... Figure 3 As shown in (b), the proportion of high-difficulty expressions in the Seen subset is 72.05%, which is higher than... Figure 3 The 59.49% of the Unseen subset shown in (c) indicates that there are significant differences in the complexity of expressions among different subsets. Figure 4 The distribution of dominant modes in the reference expression on the Ref-AVSBench test set is shown. Statistical results indicate that... Figure 4 As shown in (a), expressions jointly dominated by audio and vision accounted for the highest proportion, reaching 66.8%, significantly higher than those dominated by vision alone, audio alone, and the "dominant modality + auxiliary modality" cases. This indicates that most reference expressions require the combined use of audio and visual information to complete target determination. Further comparative analysis... Figure 4 As shown in (b) and (c), the proportion of audio-related expressions in the Unseen subset is significantly higher than that in the Seen subset, indicating that the dependence of different data subsets on modal information differs. These results validate the rationality of the explicit recognition task difficulty and dominant modality in this invention, and provide a basis for the adaptive selection of the dominant and auxiliary agents in subsequent collaborative goal reasoning mechanisms.

[0076] In summary, this invention achieves a deep understanding of the reference expression, reliable reasoning of the target object, and closed-loop correction of the segmentation results through the collaborative action of three stages: consensus multimodal recognition, collaborative target reasoning, and reflective learning segmentation. It can be effectively applied to reference audio and video segmentation tasks in complex dynamic audio and video scenarios.

[0077] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection, characterized in that... include, Consensus Multimodal Recognition: Three different multimodal cue analysis agents are used to output task difficulty, dominant modality, and reasoning based on a reference expression. These agents then interact with each other, and a final decision-making agent obtains the final task difficulty, dominant modality, and reasoning. Task difficulty is categorized as low, medium, and high. Low difficulty corresponds to tasks where target recognition relies on a single dominant modality; medium difficulty corresponds to tasks where target recognition relies on one dominant modality and another auxiliary modality; and high difficulty corresponds to tasks where target recognition relies on two dominant modalities simultaneously. Collaborative goal reasoning: Based on the final task difficulty and dominant modality obtained in the consensus multimodal recognition stage, the visual agent, audio agent, and audio-visual agent are adaptively organized to collaboratively process video frames, synchronized audio, and reference expressions to determine the initial recognition target and reasoning. Reflective learning segmentation: The segmentation agent generates an initial mask for the video frame based on the initial recognition target obtained in the collaborative goal reasoning stage, and obtains the initial segmentation result; The inspection agent then judges the consistency between the initial segmentation result and the reference expression based on the video frame with the initial mask, the synchronized audio, and the reference expression. If they are inconsistent, the corrected target is output, and the segmentation agent generates the corrected mask for the video frame to obtain the corrected segmentation result. Then, an inspection agent is used to judge the consistency between the corrected segmentation result and the reference expression based on the video frame with the corrected mask, the synchronized audio, the reference expression, and the corrected recognition target obtained in the previous iteration. This process continues until the inspection agent judges that the results are consistent or the maximum number of iterations for segmentation is reached, thus obtaining the final segmentation result.

2. The reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection as described in claim 1, characterized in that, In the consensus-based multimodal recognition phase, the three methods for peer interaction among agents in multimodal cue analysis are as follows: Each multimodal cue analysis agent reads the task difficulty, dominant mode, and rationale output by the other two multimodal cue analysis agents, re-analyzes the reference expression, and obtains the revised task difficulty, dominant mode, and rationale.

3. The reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection as described in claim 2, is characterized in that... The final decision-making agent obtains the final task difficulty, dominant mode, and reason based on the modified task difficulty, dominant mode, and reason output by the three multimodal cues.

4. The reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection as described in claim 3, is characterized in that... In the collaborative target reasoning stage, for low-difficulty tasks, if the dominant modality is the visual modality, the visual agent receives video frames and reference expressions, and outputs the initial recognition target and reason. If the dominant mode is the audio mode, the audio agent receives the synchronized audio and reference expression, and outputs the initial recognition target and reason.

5. The reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection according to claim 3, characterized in that, In the collaborative target reasoning stage, for medium-difficulty tasks, if the dominant modality is the visual modality and the auxiliary modality is the audio modality, the audio agent first receives the synchronized audio and reference expression, and outputs a list of candidate targets and reasons; then the visual agent outputs the initial identified target and reasons based on the list of candidate targets and reasons, combined with video frames and reference expressions. If the dominant modality is the audio modality and the auxiliary modality is the visual modality, then the visual agent first receives the video frames and reference expressions, and outputs a list of candidate targets and reasons; then the audio agent, based on the list of candidate targets and reasons, combined with the synchronized audio and reference expressions, outputs the initial recognition target and reasons.

6. The reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection according to claim 3, characterized in that, In the collaborative goal reasoning phase, for high-difficulty tasks, the audio agent first receives synchronized audio and reference expressions, and outputs a list of candidate goals based on audio and the reasons; at the same time, the visual agent receives video frames and reference expressions, and outputs a list of candidate goals based on video and the reasons. Then, the audio-visual agent, based on the audio-based candidate target list and reasons and the video-based candidate target list and reasons, combined with synchronized audio, video frames and reference expressions, outputs the initial recognition target and reasons.

7. The reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection according to claim 4, 5, or 6, characterized in that, During the reflective learning segmentation phase, the segmentation agent uses the initial target as a text prompt to generate an initial mask for the video frame.

8. The reference audio-visual segmentation method based on multi-agent recognition, reasoning, and reflection according to claim 1, characterized in that, In the consensus multimodal recognition stage, the dominant modality for low-difficulty tasks is either the audio modality or the visual modality; For medium-difficulty tasks, the dominant modality is either the audio or visual modality, and the auxiliary modality is either the visual or audio modality. The dominant modalities for high-difficulty tasks are audio and visual modalities.

Images