Audio-visual cooperative abnormal behavior detection method

Abstract

The invention particularly relates to a method for detecting an audio-visual cooperative abnormal behavior. The method comprises the following steps: constructing an AVadCLIP model; inputting a test video and an audio into the AVadCLIP, extracting image and audio features by using an image encoder and an audio encoder, and extracting label features of a text by using a text encoder; fusing the image and audio features to obtain a multi-modal fusion feature; performing coarse-grained binary classification on the multi-modal fusion features to obtain an abnormal confidence coefficient; performing effective audio-visual prompt by using the multi-modal fusion feature, the abnormal confidence coefficient and the text tag feature to obtain a text tag feature; and obtaining a prediction result of the test video according to the multi-modal fusion feature and the text label feature. According to the audio-visual fusion disclosed by the embodiment of the invention, adaptive cross-modal integration is realized through lightweight parameter adaptation; on the basis of semantic correlation between audio-visual features and text tags, text embedding of key multi-modal information is dynamically enhanced, and the generalization ability of CLIP to video anomaly detection tasks is remarkably improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of video abnormal behavior detection, and in particular to a method for audio-visual collaborative abnormal behavior detection. Background Art

[0002] Video anomaly detection (VAD), a key technology in intelligent surveillance systems, aims to identify anomalous events within videos and has attracted significant research interest in recent years. However, due to the rarity of anomalous events and the high cost of manual annotation, fully supervised frameworks are difficult to deploy on a large scale. To address this issue, weakly supervised video anomaly detection (WSVAD) has emerged. It requires only video-level labels rather than detailed frame-level annotations, thus reducing the reliance on finely labeled data. Current WSVAD methods are primarily based on the multiple instance learning (MIL) framework, utilizing video-level labels for model training. These methods treat videos as collections of clips (instances) and distinguish anomalous patterns through a hard attention mechanism. Despite this, existing methods mostly rely on single-modality visual information and exhibit significant limitations in complex real-world scenes. For example, factors such as visual occlusion, extreme lighting variations, and environmental noise can render visual features unreliable or ambiguous. In this context, audio-visual fusion techniques have become increasingly important. Audio, as an important source of auxiliary information, remains robust even when visual data is impaired, making it suitable for detecting out-of-view events. In acoustically rich environments, specific anomalies such as explosions, screams, or gunshots have unique acoustic signatures, making them more discernible in the audio domain. Similarly, in low-light conditions, when visual features are degraded, audio plays an important role as a key complementary modality. Therefore, integrating audio and video modalities can not only improve the accuracy of anomaly detection systems, but also enhance their robustness in diverse and complex environments. However, existing attempts to incorporate audio into video anomaly detection, such as traditional feature splicing, have failed to fully utilize the potential of modern multimodal learning techniques, resulting in poor cross-modal integration effects. In addition, these methods ignore the intrinsic semantic association between visual and auditory modalities, which is crucial for improving anomaly detection performance. In view of the above challenges, the present invention proposes a new WSVAD framework AVadCLIP, which uses audio-visual collaborative learning to drive cross-modal alignment supported by CLIP to achieve more accurate anomaly detection. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the present invention provides a method for detecting abnormal behavior in videos, which is used to solve the problem that the prior art cannot effectively utilize multimodal information (visual, text, audio) for abnormal behavior detection.

[0004] According to an embodiment of the present disclosure, a method for detecting abnormal audio-visual coordination behavior is provided, the method comprising:

[0005] A pre-trained AVadCLIP model is constructed, wherein the AVadCLIP model includes: a frozen CLIP image encoder for extracting image features of video frames; a frozen Wav2CLIP audio encoder for extracting audio features; a frozen CLIP text encoder for extracting label features of text labels; an audio-visual fusion module that introduces a gated fusion mechanism and residual mapping to fuse the image features with the audio features to generate multimodal fusion features; and an audio-visual cue module that combines label features, multimodal fusion features, and anomaly confidence to generate text label features.

[0006] Input the test video, audio and text into the trained AVadCLIP model to extract image features, audio features and text label features respectively;

[0007] The image features and audio features are fused through the audio-visual fusion module to generate multimodal fusion features at the video frame level;

[0008] Using a binary classifier to perform coarse-grained binary classification on the multimodal fusion features to obtain anomaly confidence;

[0009] Combining the text label feature with the multimodal fusion feature and the anomaly confidence by the audio-visual prompt module to generate an enhanced text label feature;

[0010] Fine-grained multi-classification is performed based on the multimodal fusion features and the enhanced text label features to obtain an abnormal behavior prediction result of the test video.

[0011] In some exemplary embodiments, the step of extracting image features of the test video using a frozen CLIP image encoder further includes:

[0012] An image frame is extracted from the training video at every preset number of frames, and the image is input into the CLIP image encoder to extract corresponding image features.

[0013] In some exemplary embodiments, the step of extracting label features of the text label corresponding to the test video using a frozen CLIP text encoder includes:

[0014] Convert the text labels into category vectors through Tokenizer;

[0015] Concatenating the category vector with the learnable hint to generate a vector sequence;

[0016] The vector sequence is encoded by the CLIP text encoder to extract the label features.

[0017] In some exemplary embodiments, the step of extracting audio features corresponding to the test video using a frozen Wav2CLIP audio encoder includes:

[0018] Every preset time period, an audio segment is intercepted from the test audio, and the audio segment is input into the Wav2CLIP audio encoder to extract corresponding audio features.

[0019] In some exemplary embodiments, the step of fusing the image features and the audio features using the audio-visual fusion module to obtain a multimodal fusion feature includes:

[0020] Performing dynamic weighted fusion on the image features and audio features through a gated fusion mechanism to generate preliminary fusion features;

[0021] The preliminary fusion features are nonlinearly transformed through residual mapping to enhance feature expression capabilities and generate the multimodal fusion features.

[0022] In some exemplary embodiments, the binary classifier comprises:

[0023] Feedforward network layer, derived from the temporal Transformer encoder; fully connected layer, with 1 neuron; Sigmoid activation function, used to output anomaly confidence.

[0024] In some exemplary embodiments, the step of combining the label feature with the multimodal fusion feature and the anomaly confidence using the audio-visual prompt module to obtain a text label feature includes:

[0025] Using the anomaly confidence as an anomaly attention weight, calculating the dot product of the anomaly attention weight and the multimodal fusion feature to generate a video-level global representation;

[0026] performing a normalization operation on the video-level global representation to generate a visual cue of abnormal focus;

[0027] Calculating a similarity matrix between the abnormally focused visual cues and the label features to generate enhanced instance-level audio-visual cues;

[0028] The enhanced instance-level audio-visual cues are element-wise added to the label features, and the final instance category embedding is generated as the text label feature through a feed-forward network and residual connections.

[0029] In some exemplary embodiments, the step of performing fine-grained multi-classification based on the multimodal fusion features and the enhanced text label features to obtain an abnormal behavior prediction result for the test video includes:

[0030] The fine-grained multi-classification is to calculate the matching similarity between the multimodal fusion feature and the text label feature to generate an alignment matrix;

[0031] Based on the alignment matrix, a multi-instance learning alignment mechanism is used to generate a prediction result of the test video.

[0032] In some exemplary embodiments,

[0033] The method further includes inputting a training video into the AVadCLIP model to obtain a trained AVadCLIP model, wherein a loss function during training includes the following steps:

[0034] For the coarse-grained binary classification, a Top-K mechanism is used to select the K highest anomaly confidence values from normal and abnormal videos respectively, and their average is calculated as the video-level prediction result of the test video. The classification loss between the video-level prediction result and the true label is calculated using binary cross entropy;

[0035] For the fine-grained multi-classification, the semantic similarity between the multimodal fusion features and all category embeddings is calculated based on the alignment matrix, the average of the top K highest similarities in each row is selected to generate the alignment vector between the video and each category, and the fine-grained multi-classification prediction result is calculated using a normalized exponential function;

[0036] Based on the multi-classification prediction results, the alignment loss is calculated by combining the cross entropy loss and the focal loss to optimize the multi-classification performance of the model;

[0037] The total loss function of the AVadCLIP model is calculated based on the classification loss and the alignment loss.

[0038] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0039] In the embodiments of the present disclosure, the audio-visual fusion module is used to combine visual information and audio information, thereby enhancing the model's ability to utilize multimodal information; the introduction of the gated fusion mechanism and residual mapping effectively improves the expressive power of multimodal features; the audio-visual prompt module and the learnable prompt mechanism enable the frozen CLIP model to adapt to the task of video abnormal behavior detection; the alignment mechanism based on multi-instance learning achieves optimization under weak supervision, maximizing the retention of the knowledge of the pre-trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0041] Figure 1 A diagram showing the steps of a method for detecting abnormal behavior through audio-visual collaboration in an exemplary embodiment of the present disclosure;

[0042] Figure 2 A framework diagram showing an AVadCLIP model in an exemplary embodiment of the present disclosure is shown;

[0043] Figure 3 A flow chart showing a method for detecting abnormal behavior through audio-visual collaboration in an exemplary embodiment of the present disclosure is shown;

[0044] Figure 4 A qualitative schematic diagram of fine-grained detection on the XD-Violence dataset in an exemplary embodiment of the present disclosure is shown;

[0045] Figure 5 A qualitative schematic diagram of coarse-grained detection on the XD-Violence dataset in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0047] In addition, the accompanying drawings are merely schematic illustrations of embodiments of the present disclosure and are not necessarily drawn to scale. Like reference numerals in the figures represent like or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0048] In this example embodiment, a method for detecting abnormal behavior through audio-visual collaboration is first provided. The method includes: constructing an AVadCLIP model; inputting training video and audio into AVadCLIP for training, extracting image features and audio features using a frozen CLIP image encoder and a frozen Wav2CLIP audio encoder, and extracting text label features using a frozen CLIP text encoder; fusing image features and audio features, processing the fused features to obtain multimodal fusion features; performing coarse-grained binary classification on the multimodal fusion features to obtain abnormality confidence; using the multimodal fusion features, abnormality confidence, and text label features to perform effective audio-visual prompting to obtain text label features; obtaining prediction results for the training video based on the multimodal fusion features and text label features; inputting a test video into the trained AVadCLIP model to obtain prediction results for the test video. The audio-visual fusion of the disclosed embodiment achieves adaptive cross-modal integration through lightweight parameter adaptation; based on the semantic correlation between audio-visual features and text labels, the text embedding of key multimodal information is dynamically enhanced, significantly improving the generalization ability of CLIP for video anomaly detection tasks.

[0049] refer to Figure 1 As shown in , the method for detecting abnormal behavior in audio-visual collaboration may include: steps S101 to S106.

[0050] Step S101: constructing a pre-trained AVadCLIP model, wherein the AVadCLIP model includes a CLIP image encoder, a CLIP text encoder, a Wav2CLIP audio encoder, an audio-visual fusion module, and an audio-visual prompt module, wherein: the frozen CLIP image encoder is used to extract image features of video frames; the frozen Wav2CLIP audio encoder is used to extract audio features; the frozen CLIP text encoder is used to extract label features of text labels; the audio-visual fusion module introduces a gated fusion mechanism and residual mapping to fuse image features with audio features to generate multimodal fusion features; the audio-visual prompt module is used to combine label features, multimodal fusion features, and anomaly confidence to generate text label features;

[0051] Step S102: input the training video and audio into AVadCLIP for training, use the frozen CLIP image encoder and the frozen Wav2CLIP audio encoder to extract image features and audio features, and use the frozen CLIP text encoder to extract text label features;

[0052] The step of extracting image features of the training video using the frozen CLIP image encoder further includes:

[0053] Every preset number of frames, a frame of image is extracted from the training video, and the image is input into the CLIP image encoder to extract the corresponding image features.

[0054] Specifically, the step of extracting label features of text labels corresponding to training videos using the frozen CLIP text encoder includes:

[0055] Convert text labels into category vectors through Tokenizer;

[0056] Concatenate the category vector with the learnable hint to generate a vector sequence;

[0057] The vector sequence is encoded through the CLIP text encoder to extract label features.

[0058] Specifically, the step of extracting audio features corresponding to the training video using the frozen Wav2CLIP audio encoder includes:

[0059] Every preset time period, an audio clip is captured from the training audio and input into the Wav2CLIP audio encoder to extract the corresponding audio features.

[0060] Step S103: fusing the image features and the audio features, and processing the fused features to obtain multimodal fused features;

[0061] Specifically, the step of fusing the image features and the audio features using the audio-visual fusion module to obtain multimodal fusion features includes:

[0062] The image features and audio features are dynamically weighted and fused through the gated fusion mechanism to generate preliminary fusion features;

[0063] The preliminary fusion features are nonlinearly transformed through residual mapping to enhance the feature expression ability and generate multimodal fusion features.

[0064] Step S104: using a binary classifier to perform coarse-grained binary classification on the multimodal fusion features to obtain anomaly confidence;

[0065] Specifically, binary classifiers include:

[0066] Feedforward network layer, derived from the temporal Transformer encoder; fully connected layer, with 1 neuron; Sigmoid activation function, used to output anomaly confidence.

[0067] Step S105: Utilize multimodal fusion features, anomaly confidence, and text label features to perform effective audiovisual prompts to obtain text label features;

[0068] Specifically, the step of combining the label feature with the multimodal fusion feature and the anomaly confidence using the audio-visual prompt module to obtain the text label feature includes:

[0069] The anomaly confidence is used as the anomaly attention weight, and the dot product of the anomaly attention weight and the multimodal fusion feature is calculated to generate a video-level global representation.

[0070] Normalize the video-level global representation to generate visual cues of abnormal focus;

[0071] Compute the similarity matrix between the abnormally focused visual cues and the label features to generate enhanced instance-level audio-visual cues;

[0072] The enhanced instance-level audio-visual cues are element-wise added to the label features and passed through a feed-forward network and residual connections to generate the final instance category embedding as the text label feature.

[0073] Step S106: Perform fine-grained multi-classification based on the multimodal fusion features and the text label features to obtain prediction results for the training video.

[0074] Specifically, the step of generating the prediction result of the training video based on the multimodal fusion features and the text label features includes:

[0075] Calculate the matching similarity between multimodal fusion features and text label features to generate an alignment matrix;

[0076] Based on the alignment matrix, a multi-instance learning alignment mechanism is used to generate prediction results for the training videos.

[0077] Furthermore, the loss function in the training process specifically includes the following steps:

[0078] For the coarse-grained binary classification, a Top-K mechanism is used to select the K highest anomaly confidence values from normal and abnormal videos respectively, and their average is calculated as the video-level prediction result of the test video. The classification loss between the video-level prediction result and the true label is calculated using binary cross entropy;

[0079] For fine-grained multi-classification, the semantic similarity between the multimodal fusion features and all category embeddings is calculated based on the alignment matrix. The average of the top K highest similarities in each row is selected to generate the alignment vector between the video and each category. The fine-grained multi-classification prediction result is calculated using a normalized exponential function.

[0080] Based on the multi-classification prediction results, the alignment loss is calculated by combining the cross entropy loss and the focal loss to optimize the multi-classification performance of the model;

[0081] The total loss function of the AVadCLIP model is calculated based on the classification loss and the alignment loss.

[0082] Step S107: Input the test video into the trained AVadCLIP model to obtain a prediction result of the test video.

[0083] Through the above-mentioned audio-visual collaborative abnormal behavior detection method, the audio-visual fusion module combines visual and audio information, enhancing the model's ability to utilize multimodal information; the introduction of the gated fusion mechanism and residual mapping effectively improves the expressive power of multimodal features; the audio-visual cue module and the learnable cue mechanism enable the frozen CLIP model to adapt to the task of video abnormal behavior detection; the alignment mechanism based on multi-instance learning achieves optimization under weak supervision and maximizes the retention of the knowledge of the pre-trained model.

[0084] This application provides a new paradigm for abnormal behavior detection, which significantly improves the accuracy and robustness of abnormal behavior detection by integrating audio-visual information and utilizing advanced multimodal learning technology. Specifically, the present invention proposes the AVadCLIP framework, such as Figure 2 As shown in the figure, this is a video anomaly detection method based on a weakly supervised learning mechanism. It not only solves the limitations of traditional single-modal detection methods (such as relying only on visual information) in complex environments, but also enhances the model's ability to identify various abnormal events by introducing information from the important dimension of audio.

[0085] The effects of this application are further described below in conjunction with simulation experiments.

[0086] 1. Simulation conditions:

[0087] The simulations for this application were performed using PyTorch software on a server containing an NVIDIA RTX4090 graphics card.

[0088] 2. Simulation content and result analysis:

[0089] Fourteen methods of the present application and the prior art (including two semi-supervised methods and twelve weakly supervised methods) were used for comparative simulation on the commonly used XD-Violence dataset.

[0090] The simulation data used in this application is XD-Violence, which is currently the largest dataset, with a total length of 217 hours. It contains 4,754 videos from six anomaly categories, of which the training set contains 3,954 videos and the test set contains 800 videos.

[0091] Table 1 Comparison results between the proposed method and existing methods on the XD-Violence dataset

[0092]

[0093]

[0094] The present invention and existing semi-supervised and weakly supervised video abnormal behavior detection methods were simulated under the same experimental settings and experimental environment, and the quantitative evaluation of the XD-Violence dataset and the comparison of abundance values were obtained as shown in Table 1. As can be seen from Table 1, the present invention method is significantly better than the semi-supervised and weakly supervised methods. More precisely, the present invention method achieved an average accuracy of 86.04% in XD-Violence, which is 1.5% and 3.6% better than the best competitors VadCLIP and DMU. Figure 4 As shown in, it is a qualitative diagram of fine-grained detection on the XD-Violence dataset; Figure 5 The figure shows a qualitative diagram of coarse-grained detection on the XD-Violence dataset.

[0095] Through the above-mentioned audio-visual collaborative abnormal behavior detection method, the audio-visual fusion module combines visual and audio information, enhancing the model's ability to utilize multimodal information; the introduction of the gated fusion mechanism and residual mapping effectively improves the expressive power of multimodal features; the audio-visual cue module and the learnable cue mechanism enable the frozen CLIP model to adapt to the task of video abnormal behavior detection; the alignment mechanism based on multi-instance learning achieves optimization under weak supervision and maximizes the retention of the knowledge of the pre-trained model.

[0096] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0097] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A method for detecting abnormal behavior by audio-visual fusion, characterized in that: The following steps are involved: A pre-trained AVadCLIP model is constructed, wherein the AVadCLIP model includes: a frozen CLIP image encoder for extracting image features of video frames; a frozen Wav2CLIP audio encoder for extracting audio features; a frozen CLIP text encoder for extracting label features of text labels; an audio-visual fusion module that introduces a gated fusion mechanism and residual mapping to fuse the image features with the audio features to generate multimodal fusion features; and an audio-visual cue module that combines label features, multimodal fusion features, and anomaly confidence to generate text label features. Input the test video, audio and text into the trained AVadCLIP model to extract image features, audio features and text label features respectively; The image features and audio features are fused through the audio-visual fusion module to generate multimodal fusion features at the video frame level; Using a binary classifier to perform coarse-grained binary classification on the multimodal fusion features to obtain anomaly confidence; Combining the text label feature with the multimodal fusion feature and the anomaly confidence by the audio-visual prompt module to generate an enhanced text label feature; Fine-grained multi-classification is performed based on the multimodal fusion features and the enhanced text label features to obtain an abnormal behavior prediction result of the test video.

2. The method for detecting abnormal behavior through audio-visual collaboration according to claim 1, characterized in that: The step of extracting image features of the test video using a frozen CLIP image encoder further includes: An image frame is extracted from the training video at every preset number of frames, and the image is input into the CLIP image encoder to extract corresponding image features.

3. The method for detecting abnormal behavior in video according to claim 2, characterized in that: The step of extracting label features of the text labels corresponding to the test video using the frozen CLIP text encoder includes: Convert the text labels into category vectors through Tokenizer; Concatenating the category vector with the learnable hint to generate a vector sequence; The vector sequence is encoded by the CLIP text encoder to extract the label features.

4. The method for detecting abnormal behavior through audio-visual collaboration according to claim 3, characterized in that: The step of extracting audio features corresponding to the test video using a frozen Wav2CLIP audio encoder includes: Every preset time period, an audio segment is intercepted from the test audio, and the audio segment is input into the Wav2CLIP audio encoder to extract corresponding audio features.

5. The method for detecting abnormal behavior in audio-visual collaboration according to claim 4, characterized in that: The step of using the audio-visual fusion module to fuse the image features and the audio features to obtain multimodal fusion features includes: Performing dynamic weighted fusion on the image features and audio features through a gated fusion mechanism to generate preliminary fusion features; The preliminary fusion features are nonlinearly transformed through residual mapping to enhance feature expression capabilities and generate the multimodal fusion features.

6. The method for detecting abnormal behavior in audio-visual collaboration according to claim 5, characterized in that: The binary classifier includes: Feedforward network layer, derived from the temporal Transformer encoder; fully connected layer, with 1 neuron; Sigmoid activation function, used to output anomaly confidence.

7. The method for detecting abnormal behavior in audio-visual collaboration according to claim 6, characterized in that: The step of combining the label feature with the multimodal fusion feature and the anomaly confidence using the audio-visual prompt module to obtain a text label feature includes: Using the anomaly confidence as an anomaly attention weight, calculating the dot product of the anomaly attention weight and the multimodal fusion feature to generate a video-level global representation; performing a normalization operation on the video-level global representation to generate a visual cue of abnormal focus; Calculating a similarity matrix between the abnormally focused visual cues and the label features to generate enhanced instance-level audio-visual cues; The enhanced instance-level audio-visual cues are element-wise added to the label features, and the final instance category embedding is generated as the text label feature through a feed-forward network and residual connections.

8. The method for detecting abnormal behavior by audio-visual fusion according to claim 7, characterized in that: The step of performing fine-grained multi-classification based on the multimodal fusion features and the enhanced text label features to obtain an abnormal behavior prediction result of the test video includes: The fine-grained multi-classification is to calculate the matching similarity between the multimodal fusion feature and the text label feature to generate an alignment matrix; Based on the alignment matrix, a multi-instance learning alignment mechanism is used to generate a prediction result of the test video.

9. The method for detecting abnormal behavior in audio-visual collaboration according to claim 1, characterized in that: The method further includes inputting a training video into the AVadCLIP model to obtain a trained AVadCLIP model, wherein a loss function during training includes the following steps: For the coarse-grained binary classification, a Top-K mechanism is used to select the K highest anomaly confidence values from normal and abnormal videos respectively, and their average is calculated as the video-level prediction result of the test video. The classification loss between the video-level prediction result and the true label is calculated using binary cross entropy; For the fine-grained multi-classification, the semantic similarity between the multimodal fusion features and all category embeddings is calculated based on the alignment matrix, the average of the top K highest similarities in each row is selected to generate the alignment vector between the video and each category, and the fine-grained multi-classification prediction result is calculated using a normalized exponential function; Based on the multi-classification prediction results, the alignment loss is calculated by combining the cross entropy loss and the focal loss to optimize the multi-classification performance of the model; The total loss function of the AVadCLIP model is calculated based on the classification loss and the alignment loss.

Images