Multi-agent collaborative control method for multi-modal audio-video processing

Abstract

This invention discloses a multi-agent cooperative control method for multimodal audio and video processing, relating to the field of cooperative control technology. The method includes: analyzing and processing video frame sequences and audio feature sequences based on timestamp alignment using each agent; constructing video causal graphs and audio causal graphs; eliminating the influence of confounding factors through counterfactual reasoning; extracting debiased video and audio features; and using a cross-modal attention mechanism for spatiotemporal alignment to generate a unified audio-video fusion feature vector set. Based on the audio-video fusion feature vector set, tasks are assigned to each agent through distributed reinforcement learning, and a self-attention mechanism is used for deep fusion to obtain a global semantic representation vector. This invention calculates relevance weights through a cross-modal attention mechanism to generate a unified audio-video fusion feature vector set, enhancing dynamic correlation between modalities and improving semantic consistency.

Description

Technical Field

[0001] This invention relates to the field of collaborative control technology, and in particular to a multi-agent collaborative control method for multimodal audio and video processing. Background Technology

[0002] The integration of multimodal audio and video processing with multi-agent cooperative control technology has made significant progress in recent years in fields such as intelligent surveillance, video conferencing, and autonomous driving. With the rapid development of deep learning and distributed computing, video feature extraction based on convolutional neural networks, audio signal processing based on recurrent neural networks or transformers, and modality fusion methods based on attention mechanisms have been widely applied to multimodal data analysis. In existing technologies, video processing typically uses object detection algorithms such as YOLO or Faster-R-CNN to extract spatial features, while audio processing relies on techniques such as Mel-frequency cepstral coefficients or WaveNet to extract temporal features. Modality fusion often employs early fusion, feature-level fusion, or decision-level fusion strategies. Multi-agent systems achieve task allocation and cooperation through distributed reinforcement learning or game theory methods. Typical frameworks such as MADDPG and PettingZoo have demonstrated efficient parallel computing capabilities in distributed audio and video processing. Furthermore, the rise of federated learning technology provides privacy-preserving and distributed optimization solutions for model updates among multiple agents. These technologies have driven the application of multimodal audio and video processing in complex scenarios.

[0003] However, existing control methods still have room for improvement. First, the spatiotemporal asynchrony of audio and video data makes feature alignment difficult. Traditional fusion methods are unable to effectively capture the dynamic correlation between modalities, reducing semantic consistency. Moreover, the interference of confounding factors on feature extraction is not fully eliminated, affecting the processing accuracy in complex scenarios. In addition, the task allocation of multi-agents in dynamic scenarios relies on static rules or predefined strategies, which are difficult to adapt to the real-time changing environmental requirements and have low computational resource allocation efficiency. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a multi-agent cooperative control method for multimodal audio and video processing to solve the problem of not eliminating confounding factors that affect processing accuracy.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a multi-agent cooperative control method for multimodal audio and video processing, comprising:

[0008] Capture video frame sequences and audio streams, and preprocess them to obtain timestamp-aligned video frame sequences and audio feature sequences;

[0009] Based on timestamp-aligned video frame sequences and audio feature sequences, each agent performs analysis and processing separately, constructs video causal graphs and audio causal graphs, eliminates the influence of confounding factors through counterfactual reasoning, extracts biased video and audio features, and uses a cross-modal attention mechanism for spatiotemporal alignment to generate a unified audio-video fusion feature vector set;

[0010] Based on the audio and video fusion feature vector set, tasks are assigned to each agent through distributed reinforcement learning, and a self-attention mechanism is used to perform deep fusion to obtain a global semantic representation vector;

[0011] Based on the global semantic representation vector, each agent makes local decisions in parallel, and uses graph neural networks to optimize the cooperation between each agent. Each agent is updated through federated learning, low-value features are eliminated, and real-time control instructions are output.

[0012] As a preferred embodiment of the multi-agent cooperative control method for multimodal audio and video processing described in this invention, the method involves: capturing video frame sequences and audio streams, and performing preprocessing to obtain timestamp-aligned video frame sequences and audio feature sequences. Specifically,

[0013] The system acquires environmental images to form a raw video stream, and simultaneously acquires environmental sound wave signals to form a raw audio stream. It assigns timestamps to the raw video stream and the raw audio stream to form a video frame sequence and an audio stream.

[0014] Gaussian filtering and pixel value normalization are applied to the video frame sequence to generate a normalized video frame sequence. Short-time Fourier transform and Mel-scale filtering are applied to the audio stream to form an audio feature vector sequence.

[0015] The normalized video frame sequence and the audio feature vector sequence are matched according to the timestamp, and the timestamp-aligned video frame sequence and audio feature sequence are output.

[0016] As a preferred embodiment of the multi-agent cooperative control method for multimodal audio and video processing described in this invention, the timestamp-aligned video frame sequence and audio feature sequence are processed separately by each agent to construct a video causal graph and an audio causal graph, specifically...

[0017] The video agent uses a pre-trained YOLOv5 model to analyze timestamp-aligned video frame sequences, identify target objects, background occlusion areas, and areas of light change, builds a video causal graph, and sets causal edges pointing from background occlusion and light change to the target object.

[0018] The audio agent uses a pre-trained Conformer model to analyze timestamp-aligned audio feature sequences, identify abnormal sounds, background noise, and echo components, build an audio causal graph, and set causal edges from background noise and echo to abnormal sounds.

[0019] As a preferred embodiment of the multi-agent cooperative control method for multimodal audio and video processing described in this invention, the step of eliminating the influence of confounding factors and extracting biased video and audio features through counterfactual reasoning specifically involves:

[0020] The video agent uses the DoWhy library to perform counterfactual reasoning on the video causal graph, fixes the true state of the target object, intervenes to change the background occlusion to an unoccluded state and the lighting changes to standard lighting conditions, and generates a debiased video feature vector sequence.

[0021] The audio agent fixes the true state of abnormal sounds on the audio causal graph, intervenes in background noise and echo, and generates a debiased audio feature vector sequence.

[0022] As a preferred embodiment of the multi-agent cooperative control method for multimodal audio and video processing described in this invention, the step of utilizing a cross-modal attention mechanism for spatiotemporal alignment to generate a unified audio and video fusion feature vector set specifically involves:

[0023] The debiased video feature vector sequence is used as the query, and the debiased audio feature vector sequence is used as the key and value. The relevance weights in the time dimension are calculated through a cross-modal attention mechanism, and the debiased audio feature vector sequence is weighted and fused to generate an enhanced audio representation.

[0024] The enhanced audio representation and the corresponding debiased video feature vector are concatenated through a linear transformation to generate a unified feature vector, which is then arranged in the order of the original timestamps to form a unified audio-video fusion feature vector set.

[0025] As a preferred embodiment of the multi-agent cooperative control method for multimodal audio and video processing described in this invention, the step of assigning tasks to each agent based on the audio and video feature vector set through distributed reinforcement learning includes:

[0026] Receive a unified set of audio and video fusion feature vectors, initialize the local policy networks of the video agent, audio agent, and fusion agent, use the task execution feedback of each agent as the reward signal, update the global policy network using the MADDPG algorithm, and generate task allocation instructions;

[0027] The task allocation instructions include a fusion agent performing a multi-source feature fusion task, a video agent performing a target tracking task, and an audio agent performing an abnormal sound classification task.

[0028] As a preferred embodiment of the multi-agent cooperative control method for multimodal audio and video processing described in this invention, wherein: the video agent performs a target tracking task and the audio agent performs an abnormal sound classification task, specifically,

[0029] The video agent uses a pre-trained DeepSORT model to generate target tracking features based on biased video feature vectors from a unified audio-video fusion feature vector set.

[0030] The audio agent uses a pre-trained PANNs classification model to generate abnormal sound classification features based on enhanced audio representations in a unified audio-video fusion feature vector set.

[0031] As a preferred embodiment of the multi-agent cooperative control method for multimodal audio and video processing described in this invention, the method of obtaining a global semantic representation vector through deep fusion using a self-attention mechanism specifically involves...

[0032] The fusion agent encodes target tracking features into target dynamic information vectors and abnormal sound classification features into sound event information vectors. These are then concatenated with unified feature vectors from a unified audio-visual fusion feature vector set to generate a multi-source feature sequence.

[0033] The multi-source feature sequence is input into the attention mechanism center, the association weights of the multi-source feature vectors are calculated, a weighted context feature sequence is generated, and the global semantic representation vector is obtained by average pooling in the time dimension.

[0034] As a preferred embodiment of the multi-agent cooperative control method for multimodal audio and video processing described in this invention, the method involves: each agent making local decisions in parallel based on a global semantic representation vector, and using a graph neural network to optimize cooperation between agents. Specifically,

[0035] The global semantic representation vector is distributed to the video agent, audio agent, and fusion agent. The video agent generates target type and trajectory information, and the audio agent generates sound event classification results.

[0036] By using video agents, audio agents, and fusion agents as nodes in a graph neural network, a fully connected agent communication graph is constructed. Through a message passing mechanism, target type and trajectory information and sound event classification results are exchanged to generate collaborative optimization parameters and adjust the data processing behavior of each agent.

[0037] As a preferred embodiment of the multi-agent cooperative control method for multimodal audio and video processing described in this invention, the step of updating each agent through federated learning, eliminating low-value features, and outputting real-time control commands specifically involves:

[0038] Each agent uploads model parameters to the federated learning aggregation server, uses the FedAvg algorithm to generate global model parameters, and distributes them to each agent to replace local model parameters.

[0039] Each agent calculates the feature activation frequency in its local feature space based on the global semantic representation vector, marks features with a feature activation frequency lower than the frequency threshold as low-value features and removes them, thus completing the elimination of low-value features.

[0040] The fusion agent generates real-time control commands based on target type and trajectory information, sound event classification results, and collaborative optimization parameters.

[0041] The beneficial effects of this invention are as follows: By combining video causal graphs and audio causal graphs with counterfactual reasoning from the DoWhy library, a debiased video feature vector sequence and a debiased audio feature vector sequence are generated, eliminating interference and improving the accuracy of feature representation. Furthermore, a unified audio-video fusion feature vector set is generated by calculating relevance weights through a cross-modal attention mechanism, enhancing dynamic association between modalities and improving semantic consistency. In addition, tasks are dynamically allocated through distributed reinforcement learning to optimize the collaboration efficiency of video agents, audio agents, and fusion agents. A communication graph of agents is constructed through graph neural networks to exchange target type and trajectory information and sound event classification results, further improving collaboration efficiency and significantly enhancing the accuracy and real-time performance of abnormal event detection in complex scenarios. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a multi-agent cooperative control method for multimodal audio and video processing.

[0044] Figure 2 Flowchart for causal counterfactual reasoning and video feature debiasing.

[0045] Figure 3 For cross-modal fusion and task allocation.

[0046] Figure 4 For multi-agent collaborative control and evolution. Detailed Implementation

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0049] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0050] Reference Figures 1-4 This is one embodiment of the present invention, which provides a multi-agent cooperative control method for multimodal audio and video processing, including the following steps:

[0051] S1: Capture video frame sequences and audio streams, and preprocess them to obtain timestamp-aligned video frame sequences and audio feature sequences;

[0052] Specifically, the steps include the following:

[0053] S1.1: Data Collection and Timestamp Addition

[0054] The system continuously captures environmental images at a fixed frame rate using a high-definition camera to form a raw video stream, while simultaneously capturing environmental sound wave signals at a fixed sampling rate using a microphone array to form a raw audio stream.

[0055] While the camera captures each frame of image data, the current time value of the unified hardware clock is appended as a timestamp to the current frame image data to generate a timestamped image. While the microphone array captures each segment of audio data, the corresponding start time value of the unified hardware clock is appended as a timestamp to the current segment of audio data to generate a timestamped audio segment. All timestamped images are arranged in timestamp order to form a video frame sequence, and all timestamped audio segments are spliced ​​together in timestamp order to form an audio stream.

[0056] S1.2: Data Preprocessing and Feature Extraction

[0057] Gaussian filtering is applied to each frame of the video frame sequence to suppress random noise in the image. Specifically, Gaussian filtering is implemented using the OpenCV library to generate a denoised image. Then, the pixel values ​​of the denoised image are normalized to the range of 0 to 1 to form a red, green and blue pixel matrix. The red, green and blue pixel matrix of each frame retains the original timestamp and is arranged in the order of the timestamps of the video frame sequence to form a normalized video frame sequence.

[0058] The Librosa library is used to perform a short-time Fourier transform on the audio stream to extract spectral information and generate a spectrogram. Based on a Mel-scale filter bank, cepstral coefficients are calculated for each audio segment every 20 milliseconds (a common window size for short-time Fourier transform, balancing time and frequency resolution). A 40-dimensional Mel-frequency cepstral coefficient vector is extracted as audio features; 40 dimensions are a standard dimension in audio processing. This forms a sequence of audio feature vectors, with each feature vector inheriting the original timestamp of the corresponding audio segment.

[0059] The normalized video frame sequence and audio feature vector sequence are precisely matched according to their respective timestamps. Data pairs with a timestamp deviation of more than 10 milliseconds are removed, and the timestamp-aligned video frame sequence and audio feature sequence are output. The 10 milliseconds is set based on human perception of audio and video.

[0060] S2: Based on timestamp-aligned video frame sequences and audio feature sequences, each agent performs analysis and processing separately, constructs video causal graphs and audio causal graphs, eliminates the influence of confounding factors through counterfactual reasoning, extracts debiased video and audio features, and uses cross-modal attention mechanism for spatiotemporal alignment to generate a unified audio-video fusion feature vector set;

[0061] Specifically, the steps include the following:

[0062] S2.1: Receive timestamp-aligned video frame sequences and audio feature sequences, start the video agent to process the video frame sequence, start the audio agent to process the audio feature sequence, and construct video and audio causal graphs, specifically as follows:

[0063] S2.1.1 The video agent analyzes each frame of the video frame sequence, identifies the target object, background occlusion area, and light change area, and establishes a video causal graph containing nodes of the target object, background occlusion, and light change. Causal edges pointing from background occlusion to the target object and from light change to the target object are set. The process is as follows:

[0064] The video agent uses the YOLOv5 model to analyze each frame of a video frame sequence. The YOLOv5 model needs to be pre-trained to ensure accurate identification of target objects, background occlusion regions, and areas of light variation. The training process is as follows: using the COCO dataset (containing bounding boxes of target objects, segmentation masks of background occlusion regions, and brightness labels of areas of light variation) and security monitoring scene data, divided into training, validation, and test sets, the YOLOv5 model is trained. The YOLOv5 model loads pre-trained weights, uses the PyTorch framework, and employs cross-entropy loss and IoU loss to learn the bounding boxes of target objects, the segmentation masks of background occlusion regions, and the brightness features of areas of light variation. Training is complete when the mean accuracy (mAP) for object detection reaches 0.9 and the IoU for segmentation reaches 0.85 on the validation set. The values ​​of 0.9 and 0.85 are set based on the user's expectations of the YOLOv5 model's performance and can be customized as needed during actual training. The trained YOLOv5 model is saved as a weight file for subsequent analysis tasks by the video agent.

[0065] The video agent receives a timestamp-aligned sequence of video frames and loads a pre-trained YOLOv5 model. It then analyzes each frame of the video sequence frame by frame to identify target objects, background occlusion areas, and areas of varying lighting. The specific operations are as follows:

[0066] The YOLOv5 model is used to extract convolutional features from each frame of the image, generating multi-scale feature maps.

[0067] The detection head predicts the bounding box coordinates and class probability of the target object based on multi-scale feature maps. For example, in security monitoring scenarios, it identifies people or vehicles and outputs bounding box coordinates and class labels (such as "people" or "vehicles") with a confidence level of 0.9 or higher.

[0068] The YOLOv5 model's segmentation branch performs semantic segmentation on each frame of the image, identifies background occlusion areas (such as leaves, obstacles, or walls), generates a segmentation mask, and marks the pixel range of the occlusion area.

[0069] The YOLOv5 model analyzes the brightness distribution of each frame of the image and identifies areas of light variation. Specifically, it converts the red, green, and blue pixel matrix into a brightness map, calculates the mean and variance of the brightness, marks areas of abnormal brightness (such as shadows or bright spots), and generates bounding boxes and brightness value ranges (e.g., shadow brightness is lower than the mean, and bright spot brightness is higher than the mean).

[0070] The analysis results for each frame of the image include the bounding box and category of the target object, the segmentation mask of the background occlusion area, the brightness label of the area with light change, and the original timestamp is preserved.

[0071] Based on the analysis results of each frame, the video agent uses the DoWhy library to build a video causal graph containing nodes for the target object, background occlusion, and lighting changes. The specific operations are as follows:

[0072] Create three nodes for each frame of the image: the target object node (representing the actual state of people or vehicles), the background occlusion node (representing the interference of leaves or obstacles), and the light change node (representing the interference of shadows or light spots).

[0073] Based on the analysis results, a causal edge is set from the background occlusion node to the target object node to indicate that background occlusion interferes with the accuracy of target object detection. For example, foliage occlusion causes the bounding box of a person to shift. A causal edge is set from the light change node to the target object node to indicate that light change affects the clarity of target object recognition. For example, shadows reduce the confidence of vehicle category.

[0074] Using the DoWhy library's causal graph interface, background occlusion nodes, target object nodes, lighting change nodes, and causal edges are organized into a directed acyclic graph (DAG) to generate a video causal graph. Each frame corresponds to one video causal graph, and timestamps are retained for subsequent counterfactual inference. The generated video causal graph contains information on target objects, background occlusion and lighting change nodes, and causal edges.

[0075] S2.1.2: The audio agent analyzes each audio feature vector in the audio feature sequence, identifies abnormal sounds, background noise, and echo components, and establishes an audio causal graph containing nodes for abnormal sounds, background noise, and echoes. It sets causal edges from background noise to abnormal sounds and from echoes to abnormal sounds, specifically:

[0076] The audio Conformer model is trained to ensure accurate identification of unusual sounds (such as breaking glass, footsteps), background noise (such as wind, vehicle noise), and echo components (such as indoor reflected sound). The Conformer model training process is as follows:

[0077] The audio agent uses the AudioSet dataset (containing abnormal sound classification, background noise classification, and echo intensity annotation) and security monitoring scenario data, divided into training and validation sets, to train the Conformer model. The Conformer model loads pre-trained weights and adopts the cross-entropy loss function to learn the spectral characteristics of abnormal sounds, the low-frequency characteristics of background noise, and the time delay characteristics of echoes for abnormal sound classification, background noise classification, and echo intensity regression. The accuracy of abnormal sound classification, the accuracy of background noise classification, and the mean square error of echo intensity are evaluated on the validation set. When the user-defined threshold is reached, the Conformer model weights are saved for audio causal graph construction to improve sound classification accuracy. The user-defined threshold is set by the user during training.

[0078] The audio agent receives a timestamp-aligned audio feature sequence, loads a pre-trained Conformer model, and analyzes each audio feature vector in the sequence to identify abnormal sounds, background noise, and echo components. The specific operations are as follows:

[0079] The audio agent analyzes the temporal, low-frequency, and time-delay features of audio feature vectors. It extracts temporal features through the multi-head self-attention mechanism and convolutional layers of the Conformer model, generating a spectral representation. It then predicts the category and confidence level of abnormal sounds (e.g., "glass breaking sound") through fully connected layers. Based on the spectral representation, it extracts low-frequency features from the first 20 dimensions of low-frequency Mel coefficients, generating a 64-dimensional noise feature representation. This predicts the category and confidence level of background noise (e.g., "wind sound"). Finally, it extracts time-delay features from the noise feature representation, generating a 64-dimensional time-delay feature representation. This predicts the intensity and time delay of echoes (e.g., indoor reflected sound). The analysis results for each audio feature vector include the category and confidence level of the abnormal sound, the category and confidence level of the background noise, and the echo intensity value, while retaining the original timestamp.

[0080] Based on the analysis results of audio feature vectors, the audio agent uses the DoWhy library to build an audio causal graph containing abnormal sounds, background noise, and echo nodes. Specifically:

[0081] For each audio feature vector, create three nodes: an abnormal sound node, a background noise node, and an echo node.

[0082] Based on the analysis results of the audio feature vectors, causal edges are set from background noise nodes to anomalous sound nodes to indicate that background noise interferes with the accuracy of anomalous sound classification. For example, wind noise reduces the confidence of glass breaking sound. Causal edges are set from echo nodes to anomalous sound nodes to indicate that echoes interfere with the clarity of anomalous sound recognition. For example, indoor reflected sound causes spectral distortion of footsteps.

[0083] Using the DoWhy library's causal graph interface, abnormal sound nodes, background noise nodes, echo nodes, and corresponding causal edges are organized into a directed acyclic graph to generate an audio causal graph. Each audio feature vector corresponds to an audio causal graph, and timestamps are retained for subsequent counterfactual reasoning.

[0084] S2.2: The video agent performs counterfactual reasoning on the video causal graph, fixes the true state of the target object, intervenes in background occlusion and lighting changes to unobstructed, standard lighting conditions, derives the target representation after eliminating occlusion and lighting interference, and generates a biased video feature sequence, specifically:

[0085] The video agent uses the YOLOv5 model to extract the bounding boxes and category labels of target objects, which are fixed as the baseline values ​​of the target object nodes in the video causal graph. Based on the baseline values ​​of the target object nodes, the DoWhy library is used to intervene in the background occlusion nodes to the unoccluded state and the lighting change nodes to the standard lighting conditions, generating a 256-dimensional debiased video feature vector containing the spatial location and category information of the target object. The debiased video feature sequence is arranged according to the original timestamps, improving the target detection accuracy in complex scenes.

[0086] S2.3: The audio agent performs counterfactual reasoning on the audio causal graph, using the DoWhy library to process the timestamp-aligned audio feature sequences based on the classification results of the Conformer model, generating a debiased audio feature sequence. The specific operations are as follows:

[0087] The audio agent uses the Conformer model to extract the category and confidence level of abnormal sounds, which are fixed as the baseline value of the abnormal sound nodes in the audio causal graph. Based on the baseline value of the abnormal sound nodes, the DoWhy library is used to intervene to bring the background noise nodes to a noise-free state and the echo nodes to a reflection-free condition. Based on the intervened audio causal graph and the spectral representation of the temporal features, a 128-dimensional debiased audio feature vector is generated, which contains the spectral information of the abnormal sounds and the original timestamps. The vectors are arranged in chronological order to form a debiased audio feature sequence, thereby improving the sound classification accuracy in complex scenes.

[0088] S2.4: Input the debiased video feature sequence and the debiased audio feature sequence into the cross-modal attention mechanism, using video features as the query and audio features as the key and value, to calculate the correlation weight between video features and audio features in the time dimension, specifically:

[0089] The cross-modal attention mechanism receives a 256-dimensional debiased video feature sequence and a 128-dimensional debiased audio feature sequence. It constructs a query matrix based on the debiased video feature vector and a key matrix and value matrix based on the debiased audio feature vector. Temporal relevance weights are calculated using dot product and softmax. The debiased audio feature vectors are then weighted and summed based on these temporal relevance weights to generate an enhanced audio representation. This enhanced audio representation is then concatenated with the debiased video feature vector through a linear transformation to form a unified feature vector containing video and audio fusion information. This unified feature vector sequence is arranged chronologically and outputs a unified audio-video fusion feature vector set, improving object detection accuracy and sound classification accuracy.

[0090] Preferably, the video agent and audio agent process the video causal graph and audio causal graph through counterfactual reasoning, generating a 256-dimensional debiased video feature sequence and a 128-dimensional debiased audio feature sequence. A cross-modal attention mechanism uses the debiased video feature vector as the query and the debiased audio feature vector as the key and value, generating a unified audio-video fusion feature vector set, improving the accuracy of object detection and sound classification in complex scenes. In contrast, traditional methods lack specific mechanisms to eliminate interference from background occlusion, lighting changes, background noise, and echoes, affecting the accuracy of processing in complex scenes. Therefore, this invention improves object detection accuracy and, in noisy environments, also enhances sound classification accuracy.

[0091] S3: Based on the audio and video fusion feature vector set, tasks are assigned to each agent through distributed reinforcement learning, and a self-attention mechanism is used to perform deep fusion to obtain a global semantic representation vector;

[0092] Specifically, the steps include the following:

[0093] S3.1: The distributed reinforcement learning framework receives a unified set of audio-video fusion feature vectors, initializes the local policy networks of the video agent, audio agent, and fusion agent, generates task execution actions based on the unified feature vectors, and constructs task execution feedback (including the target tracking accuracy of the video agent, the abnormal sound classification accuracy of the audio agent, and the feature fusion integrity of the fusion agent). It generates a reward signal, updates the global policy network using the MADDPG algorithm through a centralized training distributed execution architecture based on the reward signal, distributes policy parameters to each agent through the global policy network, and generates task allocation instructions (including the video agent performing target tracking tasks, the audio agent performing abnormal sound classification tasks, and the fusion agent performing multi-source feature fusion tasks), thereby improving task execution efficiency.

[0094] S3.2: The video agent executes the target tracking task according to the task assignment instructions and outputs target tracking features; the audio agent executes the abnormal sound classification task according to the task assignment instructions and outputs abnormal sound classification features; the fusion agent prepares to execute the feature fusion task, specifically:

[0095] The video agent loads the DeepSORT model according to the task allocation instructions, performs temporal correlation processing on the video feature part in the unified feature vector, matches the target identity across frames, and outputs target tracking features containing target ID, position trajectory, and motion speed. The process is as follows:

[0096] First, the DeepSORT model is pre-trained: The video agent is trained using the MOTCHalllenge dataset (containing bounding boxes and identity ID annotations for pedestrian video sequences), and is divided into training and validation sets. Based on the bounding boxes and identity IDs, a 128-dimensional appearance feature vector is generated using a ResNet-50-based appearance feature extraction network. Based on the appearance feature vectors, a triplet loss is used to minimize the feature distance of the same target and maximize the feature distance of different targets. Rank-1 accuracy and mAP are calculated on the validation set. When the Rank-1 accuracy reaches 85% and the mAP reaches 75%, the DeepSORT model weights are saved and used for the appearance feature extraction network to improve the target tracking accuracy in complex scenes. 85% is based on the COCO dataset and scene settings, and 75% is based on the bounding box prediction accuracy settings.

[0097] The video agent receives a unified set of audio-video fusion feature vectors and decomposes them into a 256-dimensional debiased video feature vector as auxiliary input. Simultaneously, it receives the original frame images with corresponding timestamps. The video agent calls a pre-trained YOLOv5 model to perform object detection on the original frame images, generating bounding boxes for the target objects. The YOLOv5 model performs convolutional feature extraction on each frame image, generating a feature map, and predicts the bounding boxes and class probabilities of the target objects using the detection head. The image regions within each bounding box in each frame image are cropped and input into the appearance feature extraction network of a pre-trained DeepSORT model, generating a 128-dimensional appearance feature vector representing the appearance information of the target object, such as clothing or vehicle color.

[0098] The video agent uses the bounding box position of the target object as the motion state input to the Kalman filter to predict the prior position of existing targets in the current frame. The prediction process is as follows: Based on the target position and velocity of the previous frame, the Kalman filter uses a state transition equation to predict the prior position of the current frame, generating a prior state vector containing predicted coordinates and velocity, which provides motion prediction basis for cross-frame target identity matching. The cosine similarity between the 128-dimensional appearance feature vector and the existing trajectory appearance features is calculated, and combined with the Mahalanobis distance between the prior state vector of the Kalman filter and the bounding box position of the current frame, a cost matrix is ​​generated for cross-frame target identity matching.

[0099] For a successfully matched target, the trajectory state is updated, including the target ID, the updated Kalman filter state vector (a posterior estimate based on the current bounding box position), and the appearance feature vector. For unmatched detections, a new tracking trajectory is initialized, and a new target ID is assigned. For tracking trajectories that are not matched for multiple consecutive frames (e.g., 5 frames), the video agent determines that the target has disappeared and removes it from the tracking list, for example, removing the trajectory with ID "003".

[0100] Based on the updated trajectory information, the displacement change of each target between consecutive frames is calculated, and the motion velocity is derived by combining the time interval. Finally, the target tracking features containing target ID, position trajectory and motion velocity are output, and the original timestamp is retained.

[0101] S3.2: The audio agent loads a pre-trained PANNs classification model according to the task assignment instructions, scores the enhanced audio representation in the unified feature vector, and outputs abnormal sound classification features containing sound category labels and classification confidence, specifically:

[0102] The training process of the PANNs classification model is as follows: The audio agent uses the AudioSet dataset (containing annotations of glass breaking sounds, footsteps, etc.) and background noise of the monitoring scene to construct an audio classification dataset, which is divided into a training set and a validation set. Based on the audio classification dataset, the audio is preprocessed into a 64-dimensional Log-Mel spectrogram and standardized. The Log-Mel spectrogram is trained using a PANNs classification model based on ResNet-26. The training adopts cross-entropy loss and label smoothing strategies and improves applicability through data augmentation to generate sound classification probabilities. Based on the sound classification probabilities, the classification accuracy and F1 score are calculated on the validation set. When the classification accuracy and F1 score reach the user-defined threshold, the PANNs classification model weights are saved, and the trained PANNs classification model is obtained.

[0103] The audio agent receives a unified set of audio-video fusion feature vectors and extracts the enhanced audio representation as input. It maps the enhanced audio representation back to a 64-dimensional Log-Mel spectrogram format, forming a 64×200 two-dimensional spectral input tensor, which is then standardized to have a mean of 0 and a standard deviation of 1 to match the input requirements of the PANNs model. The preprocessed spectral input tensor is fed into the trained PANNs classification model. The PANNs classification model extracts deep spectral features through a convolutional neural network, performs global average pooling, and then inputs them into the classification layer, outputting the raw predicted score of the sound. The raw predicted score of the output sound is normalized using the Softmax function to generate probability distributions for each category. The probability value of the category labeled in the dedicated audio classification dataset is the classification confidence. The category with the highest probability is selected as the final recognition result, generating the corresponding sound category label. The classification probability is converted into a 128-dimensional vector through an embedding layer, generating anomaly sound classification features containing sound category labels and classification confidence, while retaining the original timestamp.

[0104] S3.3: The target tracking features, abnormal sound classification features, and unified audio-visual fusion feature vector set received by the intelligent agent are fused together to form a multi-source feature sequence, specifically:

[0105] The fusion agent receives target tracking features (target ID, position trajectory, and movement speed), abnormal sound classification features (sound category label and classification confidence), and the current unified feature vector of a unified audio-video fusion feature vector set, while retaining the original timestamp. Based on the target ID, it encodes a one-hot vector and generates a 128-dimensional target dynamic information vector through linear transformation. Based on the target dynamic information vector, the sound category label is generated into a 64-dimensional embedding vector through an embedding layer and concatenated with the classification confidence to form a 128-dimensional sound event information vector. The sound event information vector is concatenated with the current unified feature vector and compressed into a 768-dimensional high-dimensional vector through linear transformation, containing multi-source information. Based on the high-dimensional vector, it is arranged according to the original timestamp to form a multi-source feature sequence, which improves the completeness of feature fusion.

[0106] S3.4: Input the multi-source feature sequence into the self-attention mechanism module, and generate a weighted context feature sequence by calculating the correlation weights between the feature vectors in the multi-source feature sequence, specifically:

[0107] A linear transformation is applied to each 768-dimensional multi-source feature vector of the multi-source feature sequence to generate a query matrix, a key matrix, and a value matrix. The dot product of the query matrix and the key matrix is ​​calculated to generate an attention score matrix. The attention score matrix is ​​then scaled, and arithmetic operations are performed on the square roots of the key vector dimensions to stabilize the gradient. The Softmax function is applied to the scaled attention score matrix, and normalization is performed along the last dimension to generate an association weight matrix, where each element represents the association strength between time steps.

[0108] The association weight matrix and the value matrix are multiplied by matrix to generate a weighted context feature sequence. In the weighted context feature sequence, each 768-dimensional vector integrates the context information of all time steps in the multi-source feature sequence to enhance the spatiotemporal correlation between target dynamic information and sound event information. To improve feature representation ability, a multi-head self-attention mechanism is adopted. The query, key and value generation and attention calculation operations are repeated for each head. The outputs of each head are concatenated to generate the final 768-dimensional weighted context feature sequence.

[0109] The weighted context feature sequence is residually concatenated with the input multi-source feature sequence to generate an intermediate feature sequence. The intermediate feature sequence is then normalized to ensure that the mean is 0 and the standard deviation is 1, thus generating the final weighted context feature sequence while retaining the original timestamp.

[0110] S3.5: Perform average pooling on the weighted context feature sequence in the time dimension to obtain a fixed-dimensional vector representation. The fixed-dimensional vector representation retains the global information of the multi-source feature sequence, integrates target dynamic information, sound event information and audio-visual fusion features, and stores the fixed-dimensional vector representation as a global semantic representation vector, representing the semantic information of the multi-source feature sequence within the entire time window.

[0111] Preferably, this invention utilizes a distributed reinforcement learning framework and a self-attention mechanism module to generate multi-source feature sequences and global semantic representation vectors based on a unified audio-video fusion feature vector set. This enables target tracking, abnormal sound classification, and deep fusion of multi-source features, effectively improving the detection accuracy and real-time performance of abnormal events. Compared to existing methods that process video or audio features separately, this method significantly improves the efficiency of multi-agent collaboration. The self-attention mechanism module deeply fuses target dynamic information, sound event information, and audio-video fusion features to generate a global semantic representation vector, which is superior to traditional feature concatenation methods and significantly improves the detection accuracy and real-time performance of abnormal events in complex scenarios.

[0112] S4: Based on the global semantic representation vector, each agent makes local decisions in parallel, uses a graph neural network to optimize the cooperation between each agent, updates each agent through federated learning, eliminates low-value features, and outputs real-time control instructions.

[0113] Specifically, the steps include the following:

[0114] S4.1: Receive the global semantic representation vector, distribute the global semantic representation vector to the video agent, audio agent and fusion agent at the same time, and start the local decision-making process of each agent;

[0115] S4.1.1: The video agent generates target type and trajectory information based on the global semantic representation vector, and the audio agent generates sound event classification results based on the global semantic representation vector, specifically:

[0116] The video agent is loaded with a multilayer perceptron, which consists of three fully connected layers, a classification head, and a regression head. A simulated monitoring dataset is used as the training set, and target type and trajectory information are labeled. The classification head uses cross-entropy loss, and the regression head uses mean squared error loss. The goal is to maximize the target type classification accuracy and minimize the trajectory prediction error. The training of the multilayer perceptron is completed when the classification accuracy and trajectory prediction error meet the user's requirements.

[0117] The 768-dimensional global semantic representation vector is input into the trained multilayer perceptron, which outputs a 128-dimensional vector representing the target type and trajectory information (e.g., the bounding box center coordinate sequence). The target type is predicted by the classification head, generating a category probability (e.g., "personnel" with a confidence of 0.9); the trajectory information is predicted by the regression head, generating a four-dimensional bounding box vector while retaining the original timestamp.

[0118] S4.1.2: The audio agent also uses a multilayer perceptron, but it is not the same as the video agent. The training data for the audio agent's multilayer perceptron is a subset of the AudioSet dataset from the S3.1 PANNs model. The audio agent's multilayer perceptron receives a standardized 768-dimensional global semantic representation vector, maps it to a 512-dimensional intermediate vector through the first fully connected layer, and then maps it to a 256-dimensional feature vector through the second fully connected layer. Based on the intermediate vector, it is mapped to a 64-dimensional output vector using a classification head. The Softmax function is applied to transform the output vector into a sound event classification result (e.g., "glass breaking sound"). The cross-entropy loss function and label smoothing strategy are used to train the audio agent to maximize classification accuracy and F1 score. When the classification accuracy and F1 score reach a user-defined threshold (e.g., classification accuracy 0.95, F1 score 0.93), the weights of the audio agent's multilayer perceptron are saved, completing the training. This improves the sound classification accuracy in complex scenes.

[0119] The 768-dimensional global semantic representation vector is input into the trained audio agent multilayer perceptron, and the output is a 64-dimensional vector representing the sound event classification result.

[0120] S4.2: Using video agents, audio agents, and fusion agents as nodes in a graph neural network, an agent communication graph is established. Through a message passing mechanism, target type and trajectory information, sound event classification results are exchanged in the graph neural network, and the state representation of each agent is updated.

[0121] The training process of the graph neural network is as follows: The graph neural network uses a simulated security monitoring dataset (containing video-audio scenes, labeled with target type, trajectory, sound category, and collaborative optimization target) and divides it into a training set and a validation set. Node features are generated based on the simulated security monitoring dataset. The node features are processed through two graph convolutional layers (output dimensions of 128, 64, and 768) to generate graph convolutional features. The mean squared error loss is used to train the graph convolutional features to maximize alarm triggering accuracy and minimize trajectory prediction error. The alarm triggering accuracy and trajectory prediction error are calculated on the validation set using the optimized graph convolutional features. When the alarm triggering accuracy reaches 90% and the trajectory prediction error is less than 10 pixels, the graph neural network weights are saved. The alarm triggering accuracy of 90% and the trajectory prediction error of less than 10 pixels are set based on user needs and can be customized in actual use.

[0122] The video agent, audio agent, and fusion agent are defined as three nodes in a graph neural network. The node features are their respective local decision results: the video agent (128-dimensional target type and trajectory information), the audio agent (64-dimensional sound event classification results), and the fusion agent (768-dimensional global semantic representation vector). A fully connected agent communication graph is constructed for the video agent, with bidirectional edges set between each pair of video agent nodes, audio agent nodes, and fusion agent nodes. The edge weights are initialized to 1, generating a 3×3 adjacency matrix.

[0123] The video agent, audio agent, and fusion agent use a pre-trained graph neural network to receive features from neighboring nodes through a message passing mechanism. A linear transformation is applied to each node feature to generate a message vector (with the same dimension as the node feature, such as 128 or 64 dimensions). The message vectors of the neighbors are then weighted and summed according to the edge weights of the adjacency matrix (1 in the example) through message aggregation to generate aggregated features and update the node state representation.

[0124] S4.3: The graph neural network generates collaborative optimization parameters based on the updated state representation, and distributes these parameters to the video agent, audio agent, and fusion agent to adjust the data processing behavior of each agent. Specifically:

[0125] The graph neural network generates collaborative optimization parameters based on the updated state representation (e.g., a weight adjustment factor of 0.7 for the video agent, 0.6 for the audio agent, and 0.5 for the fusion agent), which are then mapped to 3D vectors through a fully connected layer.

[0126] Collaborative optimization parameters are distributed to the video agent, audio agent, and fusion agent through data interfaces to adjust data processing behavior. For example, the video agent increases the priority of target tracking, and the audio agent enhances the confidence of sound classification.

[0127] S4.4: Each agent uploads its local model parameters to the federated learning aggregation server. The federated learning aggregation server averages and aggregates the uploaded model parameters to generate a global model update, specifically as follows:

[0128] The video agent and the audio agent upload multilayer perceptron parameters respectively, and the fusion agent uploads some graph neural network parameters.

[0129] The federated learning aggregation server receives the local model parameters from the video agent, audio agent, and fusion agent. It then calculates a weighted average using the FedAvg algorithm, with weights based on the amount of data each agent has (e.g., 1:1:1), to generate global model update parameters. These parameters include the updated multilayer perceptron and graph neural network parameters, with dimensions consistent with the local parameters. The global model update is then distributed to the video agent, audio agent, and fusion agent, where each agent replaces its local model parameters.

[0130] S4.5: After replacing the local model parameters, identify low-frequency features based on feature activation frequencies, mark low-frequency features as low-value features, and remove low-value features from the local feature space. Specifically:

[0131] The graph neural network processes a 768-dimensional global semantic representation vector and generates forward propagation activation values ​​through a message passing mechanism. Based on the forward propagation activation values, a local feature space is constructed by combining the 128-dimensional target type and trajectory information of the video agent and the 64-dimensional sound event classification results of the audio agent. The forward propagation activation values ​​of the past 100 time steps are statistically analyzed based on the local feature space, and the average value of each dimension is calculated to generate the feature activation frequency. Features with a feature activation frequency lower than the frequency threshold are marked as low-value features. The frequency threshold is set based on task relevance.

[0132] S4.6: The fusion agent determines the event type based on the sound event classification results, target type and trajectory information, and collaborative optimization parameters, generates alarm trigger commands or event recording commands, and outputs real-time control commands. Specifically:

[0133] The fusion agent receives target type and trajectory information from the video agent, sound event classification results from the audio agent, and collaborative optimization parameters, and maps them into a 2D output after splicing to represent trigger alarm instructions (confidence, e.g., 0.9) or record event instructions (confidence, e.g., 0.1).

[0134] The Softmax function is used to normalize the 2D output, and the instruction with the highest confidence (such as the alarm trigger instruction) is selected as the real-time control instruction.

[0135] This embodiment also provides a computer device applicable to a multi-agent cooperative control method for multimodal audio and video processing, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the multi-agent cooperative control method for multimodal audio and video processing as proposed in the above embodiment.

[0136] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0137] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the multi-agent cooperative control method for multimodal audio and video processing as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0138] In summary, this invention generates biased video and audio feature vector sequences by combining video and audio causal graphs with counterfactual reasoning from the DoWhy library. This eliminates interference and improves the accuracy of feature representation. Furthermore, it calculates relevance weights through a cross-modal attention mechanism to generate a unified audio-video fusion feature vector set, enhancing dynamic intermodal associations and improving semantic consistency. In addition, it optimizes the collaboration efficiency of video agents, audio agents, and fusion agents by dynamically allocating tasks through distributed reinforcement learning. Finally, it constructs an agent communication graph through graph neural networks to exchange target type and trajectory information and sound event classification results, further improving collaboration efficiency and significantly enhancing the accuracy and real-time performance of abnormal event detection in complex scenarios.

[0139] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-agent cooperative control method for multimodal audio and video processing, characterized in that: include, Capture video frame sequences and audio streams, and preprocess them to obtain timestamp-aligned video frame sequences and audio feature sequences; Based on timestamp-aligned video frame sequences and audio feature sequences, each agent performs analysis and processing separately, constructs video causal graphs and audio causal graphs, eliminates the influence of confounding factors through counterfactual reasoning, extracts biased video and audio features, and uses a cross-modal attention mechanism for spatiotemporal alignment to generate a unified audio-video fusion feature vector set; Based on the audio and video fusion feature vector set, tasks are assigned to each agent through distributed reinforcement learning, and a self-attention mechanism is used to perform deep fusion to obtain a global semantic representation vector; Based on the global semantic representation vector, each agent makes local decisions in parallel, and uses graph neural networks to optimize the cooperation between each agent. Each agent is updated through federated learning, low-value features are eliminated, and real-time control instructions are output.

2. The multi-agent cooperative control method for multimodal audio and video processing as described in claim 1, characterized in that: The video frame sequence and audio stream are captured and preprocessed to obtain a timestamp-aligned video frame sequence and audio feature sequence. Specifically, The system acquires environmental images to form a raw video stream, and simultaneously acquires environmental sound wave signals to form a raw audio stream. It assigns timestamps to the raw video stream and the raw audio stream to form a video frame sequence and an audio stream. Gaussian filtering and pixel value normalization are applied to the video frame sequence to generate a normalized video frame sequence. Short-time Fourier transform and Mel-scale filtering are applied to the audio stream to form an audio feature vector sequence. The normalized video frame sequence and the audio feature vector sequence are matched according to the timestamp, and the timestamp-aligned video frame sequence and audio feature sequence are output.

3. The multi-agent cooperative control method for multimodal audio and video processing as described in claim 2, characterized in that: The timestamp-aligned video frame sequence and audio feature sequence are processed separately by each agent to construct video causal graphs and audio causal graphs, specifically as follows: The video agent uses a pre-trained YOLOv5 model to analyze timestamp-aligned video frame sequences, identify target objects, background occlusion areas, and areas of light change, builds a video causal graph, and sets causal edges pointing from background occlusion and light change to the target object. The audio agent uses a pre-trained Conformer model to analyze timestamp-aligned audio feature sequences, identify abnormal sounds, background noise, and echo components, build an audio causal graph, and set causal edges from background noise and echo to abnormal sounds.

4. The multi-agent cooperative control method for multimodal audio and video processing as described in claim 3, characterized in that: The process of eliminating confounding factors and extracting biased video and audio features through counterfactual reasoning specifically involves: The video agent uses the DoWhy library to perform counterfactual reasoning on the video causal graph, fixes the true state of the target object, intervenes to change the background occlusion to an unoccluded state and the lighting changes to standard lighting conditions, and generates a debiased video feature vector sequence. The audio agent fixes the true state of abnormal sounds on the audio causal graph, intervenes in background noise and echo, and generates a debiased audio feature vector sequence.

5. The multi-agent cooperative control method for multimodal audio and video processing as described in claim 4, characterized in that: The method of using cross-modal attention mechanism for spatiotemporal alignment to generate a unified audio-video fusion feature vector set is as follows: The debiased video feature vector sequence is used as the query, and the debiased audio feature vector sequence is used as the key and value. The relevance weights in the time dimension are calculated through a cross-modal attention mechanism, and the debiased audio feature vector sequence is weighted and fused to generate an enhanced audio representation. The enhanced audio representation and the corresponding debiased video feature vector are concatenated through a linear transformation to generate a unified feature vector, which is then arranged in the order of the original timestamps to form a unified audio-video fusion feature vector set.

6. The multi-agent cooperative control method for multimodal audio and video processing as described in claim 5, characterized in that: The process of assigning tasks to each agent based on the audio and video feature vector set through distributed reinforcement learning includes: Receive a unified set of audio and video fusion feature vectors, initialize the local policy networks of the video agent, audio agent, and fusion agent, use the task execution feedback of each agent as the reward signal, update the global policy network using the MADDPG algorithm, and generate task allocation instructions; The task allocation instructions include a fusion agent performing a multi-source feature fusion task, a video agent performing a target tracking task, and an audio agent performing an abnormal sound classification task.

7. The multi-agent cooperative control method for multimodal audio and video processing as described in claim 6, characterized in that: The video agent performs target tracking and the audio agent performs abnormal sound classification. Specifically, The video agent uses a pre-trained DeepSORT model to generate target tracking features based on biased video feature vectors from a unified audio-video fusion feature vector set. The audio agent uses a pre-trained PANNs classification model to generate abnormal sound classification features based on enhanced audio representations in a unified audio-video fusion feature vector set.

8. The multi-agent cooperative control method for multimodal audio and video processing as described in claim 7, characterized in that: The method of using self-attention mechanism to perform deep fusion to obtain the global semantic representation vector is specifically as follows: The fusion agent encodes target tracking features into target dynamic information vectors and abnormal sound classification features into sound event information vectors. These are then concatenated with unified feature vectors from a unified audio-visual fusion feature vector set to generate a multi-source feature sequence. The multi-source feature sequence is input into the attention mechanism center, the association weights of the multi-source feature vectors are calculated, a weighted context feature sequence is generated, and the global semantic representation vector is obtained by average pooling in the time dimension.

9. The multi-agent cooperative control method for multimodal audio and video processing as described in claim 8, characterized in that: Based on a global semantic representation vector, each agent makes local decisions in parallel, and a graph neural network is used to optimize the cooperation between agents. Specifically, The global semantic representation vector is distributed to the video agent, audio agent, and fusion agent. The video agent generates target type and trajectory information, and the audio agent generates sound event classification results. By using video agents, audio agents, and fusion agents as nodes in a graph neural network, a fully connected agent communication graph is constructed. Through a message passing mechanism, target type and trajectory information and sound event classification results are exchanged to generate collaborative optimization parameters and adjust the data processing behavior of each agent.

10. The multi-agent cooperative control method for multimodal audio and video processing as described in claim 9, characterized in that: The process involves updating each agent through federated learning, eliminating low-value features, and outputting real-time control commands. Each agent uploads model parameters to the federated learning aggregation server, uses the FedAvg algorithm to generate global model parameters, and distributes them to each agent to replace local model parameters. Each agent calculates the feature activation frequency in its local feature space based on the global semantic representation vector, marks features with a feature activation frequency lower than the frequency threshold as low-value features and removes them, thus completing the elimination of low-value features. The fusion agent generates real-time control commands based on target type and trajectory information, sound event classification results, and collaborative optimization parameters.

Images