Audio and video content analysis method and device

Abstract

The embodiment of the present invention proposes a method and device for parsing audio and video content, which belongs to the field of deep learning. The audio and video to be analyzed are split, feature extracted and feature fused to obtain comprehensive visual features and auditory features. The auditory features and comprehensive visual features are input into a parsing algorithm obtained by optimization training using weakly supervised learning. Through the parsing algorithm, the auditory features and comprehensive visual features are modeled and perceptually predicted with respect to modality to obtain the action events contained in the audio and video to be analyzed and the category and modality to which each action event belongs. The parsing algorithm of the present application proposes a modality perception module and a timing perception module, which can coordinate modality and timing for evidence mining, thereby greatly reducing the model's sensitivity to pseudo-label noise generated during modal classification, improving the robustness of modality dependency judgment and the accuracy of timing annotation, and overcoming the uncertainty problem caused by the lack of timing annotation under weak supervision settings.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a method and device for analyzing audio and video content. Background Art

[0002] Supervised learning-based video parsing plays a key role in the development of smart security systems. However, it relies heavily on manually labeled multimodal video data, which increases development costs and the complexity of data analysis. Weakly supervised audio and video content parsing technology can analyze events in videos without temporal boundaries or modal annotations, labeling them as visual, auditory, or combined audiovisual events. This allows for preliminary annotation of massive amounts of coarse-grained audio videos, effectively reducing the manual effort required for data annotation.

[0003] However, current weakly supervised audio and video content analysis suffers from the pseudo-label noise sensitivity of modality classification and the uncertainty of temporal annotation. Summary of the Invention

[0004] In view of this, the object of the present invention is to provide an audio and video content analysis method and device, which can greatly reduce the pseudo-label noise sensitivity of modal classification and improve the accuracy of time series annotation.

[0005] In order to achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for parsing audio and video content, the method comprising:

[0007] Acquire the audio and video to be analyzed, split and extract features of the audio and video to be analyzed, and obtain feature information; wherein the feature information includes 2D visual features, 3D visual features, and auditory features;

[0008] fusing the 2D visual features and the 3D visual features to obtain a comprehensive visual feature;

[0009] Inputting the auditory features and the comprehensive visual features into a pre-optimized parsing algorithm; wherein the parsing algorithm is an parsing algorithm obtained by optimization training using weakly supervised learning;

[0010] Through the analysis algorithm, the auditory features and the comprehensive visual features are modeled and perceptually predicted with respect to modality to obtain analysis results; wherein, the analysis results include the action events contained in the audio and video to be analyzed and the category and modality to which each action event belongs, and the modalities include vision and hearing.

[0011] In a possible implementation, the parsing algorithm includes a modal interaction modeling module, a modal perception module, and a temporal perception module;

[0012] The step of performing modality modeling and perceptual prediction on the auditory features and the comprehensive visual features by the analytical algorithm to obtain analytical results includes:

[0013] The modal interaction modeling module performs inter-modal modeling and intra-modal modeling on the auditory feature and the comprehensive visual feature to obtain feature information; the feature information includes a single visual feature, a first mixed feature, a single audio feature, and a second mixed feature, wherein the first mixed feature is a visually guided mixed feature and the second mixed feature is an audio-guided mixed feature;

[0014] The modality perception module performs modality perception prediction on the feature information to obtain a modality prediction result; wherein the modality prediction result includes multiple action events and the modality to which each action event belongs;

[0015] The first mixed feature and the second mixed feature are sensed and predicted by the time series perception module to obtain a time series prediction result; wherein the time series prediction result includes multiple categories of action events and the occurrence time of each action event;

[0016] The modal prediction results and the time series prediction results are combined and duplicates are deleted to obtain an analytical result.

[0017] In one possible implementation, the method further includes the step of training and obtaining an optimized parsing algorithm, including:

[0018] Obtaining training set data and an initial parsing algorithm; wherein the initial parsing algorithm includes a modal interaction modeling module, a modal perception module, and a temporal perception module; the training set data includes multiple labeled audio and video samples and weakly supervised annotation information for each audio and video sample, wherein the weakly supervised annotation information includes all action events contained in the audio and video samples;

[0019] Splitting and feature extraction of each audio and video sample of the training set data to obtain comprehensive visual features and auditory features of each audio and video sample;

[0020] Extracting audio and video samples from the training set data to obtain training samples;

[0021] Using a label refinement algorithm, combining the comprehensive visual features and the auditory features, to perform modality-specific label refinement to obtain sample annotated data; wherein the sample annotated data includes audio annotated data and video annotated data;

[0022] The modal interaction modeling module performs inter-modal modeling and intra-modal modeling on the auditory features and the comprehensive visual features to obtain sample feature information, and calculates the audio-visual contrast loss based on the comprehensive visual features of the training samples and the sample feature information; wherein the sample feature information includes a single visual feature, a first mixed feature, a single audio feature, and a second mixed feature, the first mixed feature being a visually guided mixed feature, and the second mixed feature being an audio-guided mixed feature;

[0023] The modality perception module performs modality perception on the sample feature information to obtain a modality perception result, and based on the modality perception result and the sample annotation data, calculates the unimodal classification loss of the visual modality and the unimodal classification loss of the auditory modality; wherein the modality perception result includes a modality perception evidence value, an audio attention aggregation matrix, a visual attention aggregation matrix, and a probability value of each of the action events belonging to the auditory or visual mode;

[0024] The first mixed feature and the second mixed feature are perceived by the temporal perception module to obtain a temporal perception result, and a mixed modal classification loss is calculated based on the temporal perception result and the sample annotation data; wherein the temporal perception result includes a temporal perception evidence value, a temporal attention aggregation matrix, and a probability value of each category of action events;

[0025] Performing attention mechanism alignment according to the audio attention aggregation matrix, the visual attention aggregation matrix, and the temporal perception evidence value, and calculating a foreground and background consistency loss value;

[0026] The initial parsing algorithm is optimized by combining the audio-visual contrast loss, the unimodal classification loss of the visual modality, the unimodal classification loss of the auditory modality, the mixed modality classification loss and the foreground-background consistency loss value to obtain an optimized parsing algorithm.

[0027] In one possible implementation, the step of optimizing the initial parsing algorithm by combining the audio-visual contrast loss, the unimodal classification loss of the visual modality, the unimodal classification loss of the auditory modality, the mixed modality classification loss, and the foreground-background consistency loss value to obtain an optimized parsing algorithm includes:

[0028] Adding the audio-visual contrast loss, the unimodal classification loss of the visual modality, the unimodal classification loss of the auditory modality, the mixed modality classification loss, and the foreground-background consistency loss value to obtain a target loss;

[0029] According to the target loss, an optimization algorithm is used to optimize the parameters of the initial parsing algorithm.

[0030] In one possible implementation, the modal interaction modeling module includes a cross-modal attention interaction mechanism and a self-attention interaction mechanism;

[0031] The step of performing inter-modal modeling and intra-modal modeling on the auditory features and the comprehensive visual features by the modal interaction modeling module to obtain sample feature information includes:

[0032] The comprehensive visual features and the auditory features are respectively used as inputs of the cross-modal attention interaction mechanism, and the cross-modal attention mechanism processes them to obtain a first hybrid feature guided by vision and a second hybrid feature guided by audio;

[0033] The comprehensive visual features and the auditory features are respectively used as inputs of the self-attention interaction mechanism, and are processed by the self-attention interaction mechanism to obtain single visual features and single audio features.

[0034] In one possible implementation, the modality-aware module includes a mixed-modality fully-connected layer, a modality-aware fully-connected layer, and an additive attention mechanism;

[0035] The step of performing modal perception on the sample feature information by the modal perception module to obtain a modal perception result includes:

[0036] Inputting the first mixed feature and the second mixed feature into the mixed modality fully connected layer respectively, and calculating a first mixed modality perceptual evidence value for the visual segment and a second mixed modality perceptual evidence value for the audio segment;

[0037] Using the additive attention mechanism and the modality-aware fully connected layer, aggregating all of the first mixed modality perception evidence values ​​to obtain a visual mixed modality perception evidence value, and aggregating all of the second mixed modality perception evidence values ​​to obtain an audio mixed modality perception evidence value;

[0038] Inputting the single visual feature and the single audio feature into the mixed modality fully connected layer respectively, and calculating the mixed modality perceptual evidence value of the visual segment and the mixed modality perceptual evidence value of the audio segment;

[0039] Using the additive attention mechanism and the modality-aware fully connected layer, aggregating the mixed-modality perception evidence values ​​of all the visual segments to obtain video unimodal perception evidence values ​​and a visual attention aggregation matrix, and aggregating the mixed-modality perception evidence values ​​of all the audio segments to obtain audio unimodal perception evidence values ​​and an audio attention aggregation matrix;

[0040] Based on the mixed modality perception evidence value of the visual segment and the mixed modality perception evidence value of the audio segment, the probability value of the auditory modality containing each category of action event and the probability value of the visual modality containing each category of action event are calculated.

[0041] In one possible implementation, the temporal perception module includes a foreground perception fully connected layer, a background perception fully connected layer, an audio mapping fully connected layer, a visual mapping fully connected layer, a temporal perception fully connected layer, and an additive attention mechanism;

[0042] The step of perceiving the first mixed feature and the second mixed feature by the time series perception module to obtain a time series perception result includes:

[0043] The first mixed feature and the second mixed feature are perceived by the foreground perception fully connected layer, the audio mapping fully connected layer, and the visual mapping fully connected layer to obtain a foreground segment perception evidence value;

[0044] The first mixed feature and the second mixed feature are perceived by the background perception fully connected layer, the audio mapping fully connected layer, and the visual mapping fully connected layer to obtain a background segment perception evidence value;

[0045] Adopting the additive attention mechanism and the temporal perception fully connected layer, respectively aggregating the foreground segment perception evidence value and the background segment perception evidence value to obtain a temporal attention aggregation matrix, a foreground temporal perception evidence value, and a background temporal perception evidence value;

[0046] According to the foreground temporal perception evidence value, the probability value of the action event containing each category is calculated.

[0047] In a possible implementation, the step of performing attention mechanism alignment based on the audio attention aggregation matrix, the visual attention aggregation matrix, and the temporal perception evidence value to calculate the foreground and background consistency loss value includes:

[0048] Using a preset consistency loss function, the audio attention aggregation matrix, the visual attention aggregation matrix, and the temporal perception evidence value are aligned by an attention mechanism to obtain a foreground and background consistency loss value;

[0049] The consistency loss function includes:

[0050]

[0051] Among them, L fbc1 represents the consistency loss value of the foreground and background, C represents the total number of categories of action events, and A ′ Representing the temporal attention aggregation matrix, A aCharacterize the audio attention aggregation matrix, A v Characterizing visual attention aggregation matrices.

[0052] In a possible implementation, the step of splitting and extracting features from the audio and video to be analyzed to obtain feature information includes:

[0053] Splitting the audio and video to be parsed into audio to be parsed and video to be parsed, splitting the audio to be parsed into multiple audio segments, and splitting the video to be parsed into multiple video segments;

[0054] Using a pre-trained 2D convolutional neural network and a pre-trained 3D convolutional neural network, respectively extracting features from each of the video clips to obtain 2D visual features and 3D visual features corresponding to each of the video clips;

[0055] A pre-trained VGGish network is used to extract features from each audio clip to obtain auditory features corresponding to each audio clip.

[0056] In a second aspect, an embodiment of the present application provides an audio and video content analysis device, comprising a feature extraction module, a feature fusion module, an input module, and a parsing module;

[0057] The feature extraction module is used to obtain the audio and video to be analyzed, split the audio and video to be analyzed, and extract features to obtain feature information; wherein the feature information includes 2D visual features, 3D visual features, and auditory features;

[0058] The feature fusion module is used to fuse the 2D visual features and the 3D visual features to obtain a comprehensive visual feature;

[0059] The input module is used to input the auditory features and the comprehensive visual features into a pre-optimized parsing algorithm; wherein the parsing algorithm is an parsing algorithm obtained by optimization training using weakly supervised learning;

[0060] The parsing module is used to model and predict the modality of the auditory features and the comprehensive visual features through the parsing algorithm to obtain parsing results; wherein the parsing results include the action events contained in the audio and video to be parsed and the category and modality to which each action event belongs, and the modalities include vision and hearing.

[0061] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the memory being used to store one or more programs; when the one or more programs are executed by the processor, the audio and video content parsing method as described in any possible implementation method in the first aspect is implemented.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio and video content parsing method as described in any possible implementation manner in the first aspect.

[0063] The audio and video content parsing method and device provided by the embodiment of the present invention, after obtaining the audio and video to be analyzed, split and extract features of the audio and video to be analyzed to obtain feature information such as 2D visual features, 3D visual features and auditory features, and fuse the 2D visual features and 3D visual features to obtain comprehensive visual features, so as to input the auditory features and comprehensive visual features into the parsing algorithm obtained by optimization training using weak supervision learning. Through the parsing algorithm, the auditory features and comprehensive visual features are modeled and perceptually predicted with respect to the modality to obtain the parsing results, namely the action events contained in the audio and video to be analyzed and the category and modality to which each action event belongs. Based on the parsing algorithm, collaborative modal and time series evidence mining is performed, which can greatly reduce the pseudo-label noise sensitivity of modal classification and improve the accuracy of time series annotation.

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0066] Figure 1 A schematic structural diagram of an audio and video content analysis system provided by an embodiment of the present invention is shown.

[0067] Figure 2 One of the flow charts of the audio and video content parsing method provided by an embodiment of the present invention is shown.

[0068] Figure 3 Shown Figure 2 Schematic diagram of the flow of some sub-steps of step S14.

[0069] Figure 4 Shown Figure 2 Flow chart of some sub-steps of step S15.

[0070] Figure 5 The second flowchart of the audio and video content analysis method provided by the embodiment of the present invention is shown.

[0071] Figure 6 A processing logic diagram of the audio and video content parsing method provided by an embodiment of the present invention is shown.

[0072] Figure 7 Shown Figure 5 Flow chart of some sub-steps of step S27.

[0073] Figure 8 A diagram showing an application result of the audio and video content analysis method provided in an embodiment of the present application is shown.

[0074] Figure 9 A structural diagram of the audio and video content analysis device provided in an embodiment of the present application is shown.

[0075] Figure 10 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0076] Explanation of the accompanying drawings: 1000 - audio and video content analysis system; 10 - client; 20 - analysis device; 30 - training device; 40 - audio and video content analysis apparatus; 401 - feature extraction module; 402 - feature fusion module; 403 - input module; 404 - analysis module; 50 - electronic device. DETAILED DESCRIPTION

[0077] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0078] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely intended to represent selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0079] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0080] Weakly supervised audio and video content parsing is one of the most challenging tasks in multimodal video understanding. It performs event parsing on videos without temporal boundaries and modality annotation information and marks them as visual events, auditory events, or joint audio-visual events.

[0081] Weakly supervised audio and video content analysis technology holds great promise for applications in areas such as internet information management, multimedia data analysis, and human-computer interaction. In particular, supervised learning-based video understanding plays a key role in the development of smart security systems. However, this technology relies heavily on manually labeled multimodal video data, increasing development costs and complicating data analysis. However, weakly supervised audio and video content analysis technology can perform preliminary annotation on massive amounts of coarse-grained audio videos, effectively reducing the manual effort required for data annotation.

[0082] Because weakly supervised audio and video parsing lacks accurate temporal boundaries and modal annotation information, existing techniques are prone to generating pseudo-label noise during modal classification. Furthermore, weakly supervised audio and video content parsing also suffers from the uncertainty of temporal annotations. As the volume of audio and video data continues to grow, there is an urgent need to overcome the noise sensitivity of modal classification and the inaccuracy of temporal annotations.

[0083] Based on the above considerations, the embodiment of the present application improves an audio and video content analysis method, which can improve the pseudo-label noise sensitivity of modal classification of audio and video content analysis and improve the accuracy of time series annotation.

[0084] The audio and video content analysis method provided in the embodiment of the present application can be applied to Figure 1 In the audio and video content analysis system 1000 shown, the audio and video content analysis system 1000 includes a client 10, an analysis device 20 and a training device 30. The analysis device 20 can be communicatively connected with the client 10 and the training device 30 via a network.

[0085] The training device 30 is used to train and obtain an optimized parsing algorithm.

[0086] The client 10 is used to send the audio and video to be analyzed to the analysis device 20.

[0087] The parsing device 20 is used to implement the audio and video content parsing method provided in the embodiment of the present application.

[0088] It should be noted that the parsing device 20 and the training device 30 can be the same device or separate devices. Both the parsing device 20 and the training device 30 can be, but are not limited to, server clusters, standalone servers, cloud servers, personal computers, and laptop computers. The client 10 can be, but is not limited to, personal computers, laptop computers, mobile phones, mobile terminals, and wearable portable devices.

[0089] In a possible implementation, a method for parsing audio and video content is provided, referring to Figure 2 , the audio and video content parsing method can be trained and optimized by the following steps.

[0090] S10, obtaining training set data and initial parsing algorithm.

[0091] In this embodiment, the initial parsing algorithm includes a modal interaction modeling module, a modal perception module, and a temporal perception module. The training set data may include multiple audio and video samples and weakly supervised annotation information for each audio and video sample, where the weakly supervised annotation information includes all action events contained in the audio and video sample.

[0092] It should be noted that weakly supervised annotation information only includes all action events contained in the audio and video samples, and does not include the modality and corresponding time of each action event.

[0093] S11, splitting and feature extraction are performed on each audio and video sample of the training set data to obtain comprehensive visual features and auditory features of each audio and video sample.

[0094] S12, extracting audio and video samples from the training set data to obtain training samples, and using a label refinement algorithm, combined with comprehensive visual features and auditory features, to perform modality-specific label refinement to obtain sample annotated data.

[0095] The sample annotated data may include audio annotated data and video annotated data. The audio annotated data is the data obtained by annotating each action event in the weakly supervised annotation information with a priori parameters representing the auditory modality. The video annotated data is the data obtained by annotating each action event in the weakly supervised annotation information with a priori parameters representing the visual modality. It should be understood that the values ​​of the priori parameters may vary for different action events.

[0096] S13, performing inter-modal modeling and intra-modal modeling on the auditory features and the comprehensive visual features through the modal interaction modeling module to obtain sample feature information, and calculating the audio-visual contrast loss based on the comprehensive visual features and sample feature information of the training samples.

[0097] In this embodiment, the sample feature information includes a single visual feature, a first mixed feature, a single audio feature, and a second mixed feature. The first mixed feature is a visually guided mixed feature, and the second mixed feature is an audio-guided mixed feature.

[0098] S14, performing modal perception on the sample feature information through the modal perception module to obtain the modal perception result, and calculating the unimodal classification loss of the visual modality and the unimodal classification loss of the auditory modality based on the modal perception result and the sample annotation data.

[0099] It should be noted that the modal perception results may include modal perception evidence value, audio attention aggregation matrix, visual attention aggregation matrix and the probability value of each action event belonging to auditory or visual.

[0100] S15, sensing the first mixed feature and the second mixed feature through the time series perception module to obtain a time series perception result, and calculating the mixed modality classification loss based on the time series perception result and the sample annotation data.

[0101] The temporal perception results can include temporal perception evidence values, temporal attention aggregation matrices, and probability values ​​of action events of various categories.

[0102] S16, according to the audio attention aggregation matrix, the visual attention aggregation matrix and the temporal perception evidence value, performs attention mechanism alignment and calculates the foreground and background consistency loss value.

[0103] S17, combines audio-visual contrast loss, visual modality unimodal classification loss, auditory modality unimodal classification loss, mixed modality classification loss and foreground-background consistency loss value to optimize the initial parsing algorithm.

[0104] S18, determine whether the iteration end condition is met. If not, return to step S12 and execute until the iteration end condition is met. If so, execute step S19.

[0105] The conditions for ending the iteration may be, but are not limited to: (1) the loss value is less than a preset loss threshold; (2) the number of iterations reaches a preset threshold.

[0106] S19: Use the validation set data to verify the current parsing algorithm to determine whether the current parsing algorithm achieves the optimization goal. If so, the training ends and the optimized parsing algorithm is obtained. If not, the process returns to step S12.

[0107] The configuration of step S19 can be flexibly selected. For example, any verification algorithm can be used for verification, or each audio and video sample in the verification set data can be input into the current parsing algorithm and the prediction accuracy can be statistically calculated based on the results. If the prediction accuracy reaches a preset accuracy, the optimization goal is achieved. In this embodiment, no specific limitation is made.

[0108] For step S11, each audio and video sample in the training set data can be split into audio data and video data, and the audio data can be split into multiple audio segments with a preset time sequence length, and the video data can be split into multiple video segments with a preset time sequence length. At the same time, a pre-trained 2D convolutional neural network and a 3D convolutional neural network can be used to extract the 2D visual features and 3D visual features of the video data of each audio and video sample, and a pre-trained VGGish network can be used to extract the auditory features of the audio data of each audio and video sample. The 2D visual features and 3D visual features of each audio and video sample are fused into a comprehensive visual feature through a mapping layer and a concatenation operation.

[0109] The implementation of the above step S11 is only an example and not the only limitation. In other implementations, a preset rule may be used to obtain the total visual features and auditory features of each audio and video sample.

[0110] For step S12, modality-specific label refinement refers to generating differentiated modality supervision information (i.e., adding prior parameters that characterize the modality) based on the video-level annotation data of audio and video (in this embodiment, it refers to the weakly supervised annotation information of audio and video samples), so that the original weakly supervised annotation is converted into incompletely accurate audio and video annotations, and modality-dependent pseudo-labels are constructed, i.e., audio annotation data and video annotation data.

[0111] The adopted label refining algorithm may be any label refining algorithm, for example, a label smoothing mechanism or a dynamic label noise reduction technology, which is not specifically limited in this embodiment.

[0112] The label smoothing mechanism converts video-level annotation data into non-0-1 binary multi-label annotations using an average sliding calculation of the modality's prior parameters within a defined parameter range. This generates two different sets of video-level annotation data, serving as audio and visual annotations. Given modality-agnostic video-level annotation data, the label smoothing mechanism multiplies it by two different thresholds less than 1, converting the video-level annotations into two sets of non-0-1 binary soft labels, each serving as the audio and visual annotation data.

[0113] Dynamic label denoising involves adding a hybrid attention network to the entire model training process to achieve continuous label refinement. The hybrid attention network is a weakly supervised audio and video content parsing network composed of multiple layers of self-attention, cross-modal attention, and multi-label classification layers. The results of its forward computation can be used as modality-differentiated labels.

[0114] To improve label refinement, the hybrid attention network can be updated synchronously with each parameter optimization of the parsing algorithm. The loss function of each action label obtained during the forward pass for a specific modality is observed, and the magnitude of the loss function for different modalities is reduced based on each action label. This can produce more reliable audio and visual annotations and achieve label refinement.

[0115] In step S13, the modal interaction modeling module may include a cross-modal attention interaction mechanism and a self-attention interaction mechanism. The cross-modal attention interaction mechanism processes the integrated visual features and auditory features, respectively, as inputs to obtain a visually guided first hybrid feature and an audio-guided second hybrid feature. Simultaneously, the self-attention interaction mechanism processes the integrated visual features and auditory features, respectively, as inputs to obtain a single visual feature and a single audio feature.

[0116] Assume that the auditory feature extracted in step S11 is F a , where the auditory features of the i-th audio clip are The comprehensive visual feature is represented as F v , the visual features of the i-th video clip are

[0117] The first hybrid feature guided by vision can be expressed as:

[0118]

[0119] in, represents the i-th first mixed feature, σ represents the Softmax function, and d represents the number of feature dimensions.

[0120] The second mixed feature of audio guidance can be expressed as:

[0121]

[0122] in, Characterize the i-th second mixed feature.

[0123] Single visual features can be expressed as: Single audio features can be expressed as: in, Represents the i-th single visual feature, Represents the i-th single visual feature.

[0124] The set of single audio features and single visual features can be expressed as and The sets of the first mixed feature and the second mixed feature can be expressed as and

[0125] After obtaining the single audio feature set, the single visual feature set, the first mixed feature set, and the second mixed feature set, any loss function may be used to calculate the audio-video contrast loss.

[0126] In one possible implementation, the audio-video contrast loss can be expressed as:

[0127]

[0128] Among them, τ represents the Softmax function.

[0129] Through the above method, the sample feature information of each training sample can be obtained and the corresponding audio and video contrast loss can be calculated.

[0130] In a possible implementation, for step S14, the modality perception module may include a mixed modality fully connected layer, a modality perception fully connected layer, and an additive attention mechanism. Figure 3 , step S14 can be further implemented as the following steps.

[0131] S141: Input the first mixed feature and the second mixed feature into the mixed modality fully connected layer respectively, and calculate the first mixed modality perceptual evidence value for the visual segment and the second mixed modality perceptual evidence value for the audio segment.

[0132] S142, using the additive attention mechanism and the modality-aware fully connected layer, aggregates all first mixed modality perception evidence values ​​to obtain visual mixed modality perception evidence values, and aggregates all second mixed modality perception evidence values ​​to obtain audio mixed modality perception evidence values.

[0133] The first mixed modality perceptual evidence value can be expressed as: The second mixed modality perceptual evidence value can be expressed as: Among them, δ represents the exponential function, f inter Characterizing mixed-modal fully connected layers.

[0134] The visual mixed modality perception evidence value can be expressed as:

[0135]

[0136] in, Characterize the first mixed feature matrix, R va Characterize the visual mixed attention aggregation matrix, f mod Representation modality perception fully connected layer, Characterizing the evidential value of visual mixed-modal perception.

[0137] The audio mixed modality perceptual evidence value can be expressed as:

[0138]

[0139] in, Characterize the second mixed feature matrix, A av Characterize the audio mixture attention aggregation matrix, Characterizing the perceptual evidence value of audio mixed modalities.

[0140] S143: Input the single visual feature and the single audio feature into the mixed modality fully connected layer respectively to calculate the mixed modality perception evidence value of the visual segment and the mixed modality perception evidence value of the audio segment.

[0141] S144 uses the additive attention mechanism and the modality-aware fully connected layer to aggregate the mixed-modality perception evidence values ​​of all visual clips to obtain the video unimodal perception evidence value and the visual attention aggregation matrix, and to aggregate the mixed-modality perception evidence values ​​of all audio clips to obtain the audio unimodal perception evidence value and the audio attention aggregation matrix.

[0142] The mixed modality perceptual evidence value of a visual segment can be expressed as: The mixed modality perceptual evidence value of an audio clip can be expressed as:

[0143] The video unimodal perceptual evidence value can be expressed as: in, Representing the single visual feature matrix, A v Representing the visual attention aggregation matrix, Characterizing the unimodal perceptual evidence value of videos.

[0144] The audio unimodal perception evidence value can be expressed as:

[0145] in, Characterize the single audio feature matrix, A a Characterize the audio attention aggregation matrix, Characterizing the perceptual evidence value of audio unimodality.

[0146] S145 , based on the mixed modality perception evidence value of the visual segment and the mixed modality perception evidence value of the audio segment, calculate the probability value of the auditory modality containing each category of action event and the probability value of the visual modality containing each category of action event.

[0147] In step S145 , a probability value calculation formula may be used to calculate the probability that the auditory modality or the visual modality contains an action event.

[0148] The probability value calculation formula may include: Among them, m∈{av,va}, Represents the evidence value corresponding to the action event of the i-th category in the mixed-modal perception evidence value of the visual segment, the unimodal perception evidence value of the video segment, the mixed-modal perception evidence value of the audio segment, and the unimodal perception evidence value of the audio. When m is va, Represents the probability value of the action event of the i-th category in the visual modality. When m is av, Characterizes the probability value of the auditory modality containing the action event of the i-th category.

[0149] After obtaining the modal perception results (i.e., the modal perception evidence value, the audio attention aggregation matrix, the visual attention aggregation matrix, and the probability value of each action event belonging to auditory or visual) through the above steps S141 to S145, in step S14, any loss function can be used to calculate the unimodal classification loss of the visual modality and the unimodal classification loss of the auditory modality. In this embodiment, no specific limitation is made.

[0150] In one possible implementation, the unimodal classification loss of the visual modality (also referred to as the unimodal action label classification loss of the visual modality) can be expressed as:

[0151]

[0152] in, c represents the total number of action event categories contained in the audio and video samples, Represents the visual annotation corresponding to the i-th action category in the sample annotation data, Represents the evidence value corresponding to the action event of the i-th category in the mixed modal perception evidence value of the visual segment, Represents the evidence value corresponding to the action event of the i-th category in the video unimodal perception evidence value.

[0153] The unimodal classification loss of the auditory modality (also known as the unimodal action label classification loss of the auditory modality) can be expressed as:

[0154]

[0155] in, Represents the auditory annotation corresponding to the i-th action category in the sample annotation data, Represents the evidence value corresponding to the action event of the i-th category in the mixed modal perception evidence value of the audio clip, Represents the evidence value corresponding to the action event of the i-th category in the audio unimodal perception evidence value.

[0156] In a possible implementation, the temporal perception module may include a foreground perception fully connected layer, a background perception fully connected layer, an audio mapping fully connected layer, a visual mapping fully connected layer, a temporal perception fully connected layer, and an additive attention mechanism. Figure 4 , step S15 can obtain the timing perception result through the following steps.

[0157] S151 , sensing the first mixed feature and the second mixed feature through the foreground perception fully connected layer, the audio mapping fully connected layer, and the visual mapping fully connected layer to obtain a foreground segment perception evidence value.

[0158] S152: The first mixed feature and the second mixed feature are perceived through the background perception fully connected layer, the audio mapping fully connected layer, and the visual mapping fully connected layer to obtain a background segment perception evidence value.

[0159] S153 uses the additive attention mechanism and the temporal perception fully connected layer to aggregate the foreground segment perception evidence value and the background segment perception evidence value respectively to obtain the temporal attention aggregation matrix, the foreground temporal perception evidence value and the background temporal perception evidence value.

[0160] S154, calculating the probability value of action events of each category based on the foreground temporal perception evidence value.

[0161] In step S151, the perceived evidence value of the foreground segment can be expressed as: Among them, f fore Representation of foreground-aware fully connected layer, f a Representation audio mapping fully connected layer, f v Representing visual map fully connected layers.

[0162] In step S152, the background segment perception evidence value can be expressed as: Among them, f back Characterizing the background-aware fully connected layer.

[0163] In step S153, the foreground temporal perception evidence value can be expressed as: The background temporal perception evidence value can be expressed as: in, A ′ Representing the temporal attention aggregation matrix, f mom Representing the timing-aware fully connected layer.

[0164] It should be emphasized that in the calculation of the background temporal perception evidence value, attention reversal calculation is performed through R(A′)=(1-A′).

[0165] In step S154, the probability value of the audio and video containing each type of action event can be expressed as: in, n∈{f,b}, Represents the evidence value corresponding to the action event of the i-th category in the foreground temporal perception evidence value and the background temporal perception evidence value. When n is f, The probability value that represents the audio or video contains the action event of the i-th category.

[0166] Based on the above, the mixed modality classification loss in step S15 can be calculated using any loss function, which is not specifically limited in this embodiment.

[0167] In one possible implementation, the mixed modality classification loss (also referred to as mixed modality action multi-label classification loss that does not distinguish between modal categories) of step S15 can be expressed as:

[0168]

[0169] in, Represents the i-th action category in the weakly supervised annotation information (i.e., the video-level annotation corresponding to the i-th action category that does not distinguish between modal categories).

[0170] Through the above implementation, temporal perception of audio and video (i.e., audio and video samples) is performed, and the probability values ​​of the audio and video samples containing action events of various categories are calculated, as well as the mixed-modal action multi-label classification loss that does not distinguish between modal categories.

[0171] In step S16, attention alignment can be performed using any alignment mechanism, or according to preset rules, which is not specifically limited in this embodiment.

[0172] In a possible implementation, a preset consistency loss function can be used to perform attention mechanism alignment on the audio attention aggregation matrix, the visual attention aggregation matrix, and the temporal perception evidence value to obtain a foreground and background consistency loss value.

[0173] The consistency loss function can be expressed as: Among them, L fbc1represents the consistency loss value of the foreground and background, C represents the total number of categories of action events, and A ′ Representing the temporal attention aggregation matrix, A a Representing the audio attention aggregation matrix, R v Characterizing visual attention aggregation matrices.

[0174] In one possible implementation, in step S17, the audio-visual contrast loss, the visual modality unimodal classification loss, the auditory modality unimodal classification loss, the mixed modality classification loss, and the foreground-background consistency loss can be summed to obtain a target loss. Based on the target loss, an optimization algorithm is used to optimize the parameters of the initial parsing algorithm.

[0175] The setting of the optimization algorithm can be flexibly selected. For example, it can be any optimization algorithm, or it can be optimized according to preset rules. This is not specifically limited in this embodiment.

[0176] In an optional implementation, an Adam optimizer may be used to optimize the initial parsing algorithm.

[0177] In addition, when dynamic label denoising technology is used to perform modality-specific label refinement in step S12, while the Adam optimizer is used to optimize the initial parsing algorithm, an Adam optimizer can be set up separately for the hybrid attention network used for label refinement to simultaneously optimize and update the hybrid attention network.

[0178] Through the above steps S10 to S19 and their sub-steps, an optimized and mature parsing algorithm can be obtained, and then the parsing algorithm can be deployed in an application environment to parse audio and video content.

[0179] In one possible implementation, refer to Figure 5 The audio and video content parsing method provided in the embodiment of the present application may further include the following steps: parsing of the audio and video content is achieved through the following steps, that is, the following steps are the audio and video content parsing process after the parsing algorithm is deployed to the parsing device 20.

[0180] S21, obtaining audio and video to be analyzed, splitting and extracting features from the audio and video to be analyzed, and obtaining feature information.

[0181] S23, fusing the 2D visual features and the 3D visual features to obtain comprehensive visual features.

[0182] It should be noted that the feature information includes 2D visual features, 3D visual features and auditory features.

[0183] S25, inputting the auditory features and the comprehensive visual features into a pre-optimized parsing algorithm.

[0184] In this embodiment, the analytical algorithm is an analytical algorithm obtained by optimizing and training using weakly supervised learning, that is, an analytical algorithm obtained by using the training method of steps S10 to S19 above.

[0185] S27, through the analytical algorithm, the auditory features and the comprehensive visual features are modeled and perceived in terms of modality to obtain analytical results.

[0186] The parsing results include the action events contained in the audio and video to be parsed, as well as the category and mode to which each action event belongs. The modes include visual and auditory.

[0187] In the above-mentioned audio and video content parsing method, after obtaining the audio and video to be parsed, the audio and video to be parsed are split and feature extracted to obtain feature information such as 2D visual features, 3D visual features and auditory features, and the 2D visual features and 3D visual features are fused to obtain comprehensive visual features, so that the auditory features and comprehensive visual features are input into the parsing algorithm obtained by optimization training using weakly supervised learning. Through the parsing algorithm, the auditory features and comprehensive visual features are modeled and perceptually predicted with respect to the modality to obtain the parsing results, namely the action events contained in the audio and video to be parsed and the category and modality to which each action event belongs. Collaborative modal and temporal evidence mining based on the parsing algorithm can greatly reduce the pseudo-label noise sensitivity of modal classification and improve the accuracy of temporal annotation.

[0188] The method for obtaining feature information in step S21 is similar to the method for obtaining feature information in step 11. The audio and video to be parsed can be split into the audio to be parsed and the video to be parsed, the audio to be parsed can be split into multiple audio segments, and the video to be parsed can be split into multiple video segments. Pre-trained 2D convolutional neural networks and 3D convolutional neural networks are used to extract features from each video segment, respectively, to obtain 2D visual features and 3D visual features corresponding to each video segment. At the same time, a pre-trained VGGish network can be used to extract features from each audio segment to obtain auditory features corresponding to each audio segment.

[0189] For step S23 , the process of obtaining the comprehensive visual features can refer to the implementation of step S11 above, and will not be repeated here.

[0190] It should be understood that the feature information may include a single visual feature, a first mixed feature, a single audio feature, and a second mixed feature, wherein the first mixed feature is a visually guided mixed feature and the second mixed feature is an audio guided mixed feature.

[0191] The parsing algorithm after training and optimization can include modal interaction modeling module, modal perception module and time sequence perception module. Figure 6 , Figure 6The audio and video content parsing logic diagram of the audio and video content parsing method provided by this application in the deployment environment. On this basis, refer to Figure 7 , step S27 can be implemented as the following steps.

[0192] S271, through the modal interaction modeling module, perform inter-modal modeling and intra-modal modeling on the auditory features and the comprehensive visual features to obtain feature information.

[0193] S272, performing modal perception prediction on the feature information through the modal perception module to obtain a modal prediction result.

[0194] S273, performing perception prediction on the first mixed feature and the second mixed feature through the time series perception module to obtain a time series prediction result.

[0195] It should be noted that steps S271 to S273 can be executed simultaneously, that is, in parallel, or can be executed sequentially in any order.

[0196] In this embodiment, the modality prediction results include the action events contained in each audio clip and each video clip, that is, multiple action events and the modality to which each action event belongs. The timing prediction results include all categories of action events present in all audio clips and all video clips. The time corresponding to each audio clip and video clip can be used to determine the occurrence event of each action category, that is, multiple categories of action events and the occurrence time of each action event.

[0197] For the audio and video clips to be parsed (i.e., clips that include both audio and video), clips with the same category action time will be merged and regarded as the same event, thereby finally completing the audio and video content parsing.

[0198] Based on the audio and video content parsing method provided above, multiple evaluation scenarios are provided to verify its sensitivity to pseudo-label noise for modal classification and the accuracy of temporal annotation.

[0199] Evaluation Scenario 1

[0200] The performance evaluation work is carried out on the Look Listen and Parse (LLP) dataset. The LLP dataset covers a wide range of daily life scenes, including human activities, animal activities, music performances, and vehicle sounds.

[0201] Detailed event categories can include a man talking, a dog barking, playing guitar, and frying food, etc. Each video with sound lasts for 10 seconds. In particular, a total of 7,202 video clips were labeled with more than one event category, with an average of 1.64 different event categories per video.

[0202] The audio and video content parsing method provided in the above embodiment was used for training, validation, and testing using standard splits of 10,000, 649, and 1,200, respectively. Based on the above data, the audio and video content parsing method provided in this application was compared with other methods on this dataset using two label refinement methods: label smoothing and dynamic label denoising. Table 1 shows a performance comparison of each algorithm on the LLP dataset using both label smoothing and dynamic label denoising.

[0203] Table 1

[0204]

[0205]

[0206] As can be seen from Table 1, the performance of the audio and video content parsing method provided by this application is better than other algorithms.

[0207] Evaluation Scenario 2

[0208] The effect evaluation work was carried out while eliminating some model structures. The parsing algorithm provided in this application still showed certain performance advantages after removing some model structures. In this embodiment, verification work was carried out on the LLP dataset. The results of each structure combination are shown in Table 2, which is a performance comparison table when removing some model structures.

[0209] Table 2

[0210]

[0211] In Table 2, ModEL represents the modality-aware fully connected layer, MomEL represents the temporal-aware fully connected layer, FBCL represents the foreground-background consistency loss, Lc represents the audio-video contrast loss, "×" represents removal, and "√" represents presence. As shown in Table 2, after eliminating some of the model structure of the parsing algorithm, the performance fluctuation of the parsing algorithm is relatively small.

[0212] When deploying the parsing algorithm provided in this application to parse audio and video content, this embodiment works well on the test data. The results of audio and video content parsing are as follows: Figure 8 shown.

[0213] Compared with the traditional weakly supervised audio and video content analysis method, the audio and video content analysis method provided by this application has the following beneficial effects:

[0214] (1) By proposing a modality-aware evidence learning module (i.e., modality-aware module), the sensitivity of the parsing algorithm to the pseudo-label noise generated during modality classification can be reduced, achieving a more robust modality dependency judgment.

[0215] (2) A time-series-aware evidence learning module (i.e., time-series-aware module) is proposed to introduce the theory of evidence deep learning into time-series modeling, thereby overcoming the uncertainty problem caused by the lack of time-series annotations under weak supervision settings.

[0216] (3) Evidence learning mechanism in two aspects: collaborative modality perception and temporal perception. A foreground-background consistency learning mechanism is designed in the training of the parsing algorithm, so that the model can conduct evidence mining in a collaborative modality and temporal manner during the learning process, and perceive the foreground part with semantic significance, thereby greatly improving the parsing ability of the parsing algorithm.

[0217] Based on the same concept as the above-mentioned audio and video content analysis method, in a possible implementation, an audio and video content analysis device 40 is further provided. The audio and video content analysis device 40 can be applied to Figure 1 The parsing device 20 in FIG. Figure 9 The audio and video content analysis device 40 may include a feature extraction module 401 , a feature fusion module 402 , an input module 403 and a parsing module 404 .

[0218] The feature extraction module 401 is used to obtain the audio and video to be analyzed, split the audio and video to be analyzed, and extract features to obtain feature information, wherein the feature information includes 2D visual features, 3D visual features, and auditory features.

[0219] The feature fusion module 402 is used to fuse the 2D visual features and the 3D visual features to obtain comprehensive visual features.

[0220] The input module 403 is used to input the auditory features and the comprehensive visual features into a pre-optimized parsing algorithm, wherein the parsing algorithm is an parsing algorithm obtained by optimization training using weakly supervised learning.

[0221] The parsing module 404 is configured to perform modality modeling and perceptual prediction on the auditory features and the comprehensive visual features using the parsing algorithm to obtain parsing results. The parsing results include action events contained in the audio or video to be parsed, as well as the category and modality to which each action event belongs, including visual and auditory modalities.

[0222] In a possible implementation, a training optimization module may also be included to train and obtain an optimized parsing algorithm.

[0223] Furthermore, the training optimization module is used to: obtain training set data and an initial parsing algorithm; split and extract features of each audio and video sample in the training set data to obtain comprehensive visual features and auditory features of each audio and video sample; extract audio and video samples from the training set data to obtain training samples; use a label refinement algorithm to combine the comprehensive visual features and the auditory features to perform modality-specific label refinement to obtain sample annotated data; perform inter-modal modeling and intra-modal modeling on the auditory features and the comprehensive visual features through the modal interaction modeling module to obtain sample feature information, and calculate the audio-visual contrast loss based on the comprehensive visual features and sample feature information of the training samples. loss; the modal perception module is used to perform modal perception on the sample feature information to obtain the modal perception result, and based on the modal perception result and the sample annotation data, the unimodal classification loss of the visual modality and the unimodal classification loss of the auditory modality are calculated; the first mixed feature and the second mixed feature are perceived by the temporal perception module to obtain the temporal perception result, and based on the temporal perception result and the sample annotation data, the mixed modal classification loss is calculated; the initial parsing algorithm is optimized by combining the audio-visual contrast loss, the unimodal classification loss of the visual modality, the unimodal classification loss of the auditory modality, the mixed modal classification loss and the foreground-background consistency loss value to obtain the optimized parsing algorithm.

[0224] In the above-mentioned audio and video content analysis device 40, through the coordinated action of the feature extraction module, the feature fusion module 402, the input module 403 and the analysis module 404, after obtaining the audio and video to be analyzed, the audio and video to be analyzed are split and feature extracted to obtain feature information such as 2D visual features, 3D visual features and auditory features, and the 2D visual features and the 3D visual features are fused to obtain comprehensive visual features, so that the auditory features and the comprehensive visual features are input into the analysis algorithm obtained by optimization training using weak supervision learning. Through the analysis algorithm, the auditory features and the comprehensive visual features are modeled and perceptually predicted with respect to the modality to obtain the analysis results, namely the action events contained in the audio and video to be analyzed and the category and modality to which each action event belongs. Based on the analysis algorithm, collaborative modal and temporal evidence mining can be performed, which can greatly improve the pseudo-label noise sensitivity of modal classification and the accuracy of temporal annotation.

[0225] For the specific definition of the audio and video content analysis device 40, please refer to the definition of the audio and video content analysis method above, and will not be repeated here. The various modules in the above-mentioned audio and video content analysis device 40 can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device 50 in hardware form, or can be stored in the memory of the electronic device 50 in software form, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0226] In one embodiment, an electronic device 50 is provided, whose internal structure diagram can be as follows: Figure 10 As shown. The electronic device 50 includes a processor, a memory, a communication interface and an input device connected via a system bus. Among them, the processor of the electronic device 50 is used to provide computing and control capabilities. The memory of the electronic device 50 includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the electronic device 50 is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, the audio and video content parsing method provided in the above embodiment is implemented.

[0227] Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the electronic device 50 to which the solution of the present invention is applied. The specific electronic device 50 may include Figure 10 More or fewer components may be shown, or some components may be combined, or the components may be arranged differently.

[0228] In one embodiment, the audio and video content analysis device 40 provided by the present invention for use in a deployed device can be implemented in the form of a computer program. The computer program can be used in a computer program such as Figure 10 The memory of the electronic device 50 can store various program modules constituting the audio and video content analysis device 40, such as: Figure 9 The feature extraction module, feature fusion module 402, input module 403 and analysis module 404 are shown. The computer program composed of various program modules enables the processor to execute the steps of the audio and video content analysis method described in this specification.

[0229] For example, Figure 10 The electronic device 50 shown may be Figure 9 The feature extraction module in the audio and video content analysis device 40 shown in FIG. 40 performs step S21. The electronic device 50 can perform step S23 through the feature fusion module 402. The electronic device 50 can perform step S25 through the input module 403. The electronic device 50 can perform step S27 through the analysis module 404.

[0230] In one embodiment, an electronic device 50 is provided, comprising: a processor and a memory, the memory being used to store one or more programs; when the one or more programs are executed by the processor, the following steps are implemented: obtaining audio and video to be analyzed, splitting and extracting features from the audio and video to be analyzed, and obtaining feature information; fusing 2D visual features and 3D visual features to obtain comprehensive visual features; inputting auditory features and the comprehensive visual features into a pre-optimized analysis algorithm; and using the analysis algorithm, modeling and perceptually predicting the auditory features and the comprehensive visual features with respect to modality to obtain analysis results.

[0231] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining audio and video to be analyzed, splitting and extracting features of the audio and video to be analyzed, and obtaining feature information; fusing 2D visual features and 3D visual features to obtain comprehensive visual features; inputting auditory features and the comprehensive visual features into a pre-optimized analysis algorithm; and using the analysis algorithm, modeling and perceptually predicting the auditory features and the comprehensive visual features with respect to modality to obtain analysis results.

[0232] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a portion of code, and the module, program segment or a portion of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0233] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0234] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or electronic device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0235] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for analyzing audio and video content, characterized in that: The method comprises: Acquire the audio and video to be analyzed, split and extract features of the audio and video to be analyzed, and obtain feature information; wherein the feature information includes 2D visual features, 3D visual features, and auditory features; fusing the 2D visual features and the 3D visual features to obtain a comprehensive visual feature; Inputting the auditory features and the comprehensive visual features into a pre-optimized parsing algorithm; wherein the parsing algorithm is an parsing algorithm obtained by optimization training using weakly supervised learning; Using the parsing algorithm, the auditory features and the comprehensive visual features are modeled and perceptually predicted with respect to modality to obtain parsing results; wherein the parsing results include action events contained in the audio or video to be parsed and the category and modality to which each action event belongs, the modalities including visual and auditory; The parsing algorithm includes a modal interaction modeling module, a modal perception module, and a temporal perception module; the modal interaction modeling module includes a cross-modal attention interaction mechanism and a self-attention interaction mechanism; the modal perception module includes a mixed modal fully connected layer, a modal perception fully connected layer, and an additive attention mechanism; the temporal perception module includes a foreground perception fully connected layer, a background perception fully connected layer, an audio mapping fully connected layer, a visual mapping fully connected layer, a temporal perception fully connected layer, and an additive attention mechanism; The step of performing modality modeling and perceptual prediction on the auditory features and the comprehensive visual features by the analytical algorithm to obtain analytical results includes: The modal interaction modeling module performs inter-modal modeling and intra-modal modeling on the auditory feature and the comprehensive visual feature to obtain feature information; the feature information includes a single visual feature, a first mixed feature, a single audio feature, and a second mixed feature, wherein the first mixed feature is a visually guided mixed feature and the second mixed feature is an audio-guided mixed feature; The modality perception module performs modality perception prediction on the feature information to obtain a modality prediction result; wherein the modality prediction result includes multiple action events and the modality to which each action event belongs; The first mixed feature and the second mixed feature are sensed and predicted by the time series perception module to obtain a time series prediction result; wherein the time series prediction result includes multiple categories of action events and the occurrence time of each action event; The modal prediction results and the time series prediction results are combined and duplicates are deleted to obtain an analytical result.

2. The audio and video content analysis method according to claim 1, characterized in that: The method further comprises the steps of training and obtaining an optimized parsing algorithm, including: Obtaining training set data and an initial parsing algorithm; wherein the initial parsing algorithm includes a modal interaction modeling module, a modal perception module, and a temporal perception module; the training set data includes multiple labeled audio and video samples and weakly supervised annotation information for each audio and video sample, wherein the weakly supervised annotation information includes all action events contained in the audio and video samples; Splitting and feature extraction of each audio and video sample of the training set data to obtain comprehensive visual features and auditory features of each audio and video sample; Extracting audio and video samples from the training set data to obtain training samples; Using a label refinement algorithm, combining the comprehensive visual features and the auditory features, to perform modality-specific label refinement to obtain sample annotated data; wherein the sample annotated data includes audio annotated data and video annotated data; The modal interaction modeling module performs inter-modal modeling and intra-modal modeling on the auditory features and the comprehensive visual features to obtain sample feature information, and calculates the audio-visual contrast loss based on the comprehensive visual features of the training samples and the sample feature information; wherein the sample feature information includes a single visual feature, a first mixed feature, a single audio feature, and a second mixed feature, the first mixed feature being a visually guided mixed feature, and the second mixed feature being an audio-guided mixed feature; The modality perception module performs modality perception on the sample feature information to obtain a modality perception result, and based on the modality perception result and the sample annotation data, calculates the unimodal classification loss of the visual modality and the unimodal classification loss of the auditory modality; wherein the modality perception result includes a modality perception evidence value, an audio attention aggregation matrix, a visual attention aggregation matrix, and a probability value of each of the action events belonging to the auditory or visual mode; The first mixed feature and the second mixed feature are perceived by the temporal perception module to obtain a temporal perception result, and a mixed modal classification loss is calculated based on the temporal perception result and the sample annotation data; wherein the temporal perception result includes a temporal perception evidence value, a temporal attention aggregation matrix, and a probability value of each category of action events; Performing attention mechanism alignment according to the audio attention aggregation matrix, the visual attention aggregation matrix, and the temporal perception evidence value, and calculating a foreground and background consistency loss value; The initial parsing algorithm is optimized by combining the audio-visual contrast loss, the unimodal classification loss of the visual modality, the unimodal classification loss of the auditory modality, the mixed modality classification loss and the foreground-background consistency loss value to obtain an optimized parsing algorithm.

3. The audio and video content analysis method according to claim 2, characterized in that: The step of optimizing the initial parsing algorithm by combining the audio-visual contrast loss, the unimodal classification loss of the visual modality, the unimodal classification loss of the auditory modality, the mixed modality classification loss, and the foreground-background consistency loss value to obtain an optimized parsing algorithm includes: Adding the audio-visual contrast loss, the unimodal classification loss of the visual modality, the unimodal classification loss of the auditory modality, the mixed modality classification loss, and the foreground-background consistency loss value to obtain a target loss; According to the target loss, an optimization algorithm is used to optimize the parameters of the initial parsing algorithm.

4. The audio and video content analysis method according to claim 2, characterized in that: The step of performing inter-modal modeling and intra-modal modeling on the auditory features and the comprehensive visual features by the modal interaction modeling module to obtain sample feature information includes: The integrated visual feature and the auditory feature are respectively used as inputs of the cross-modal attention interaction mechanism, and are processed by the cross-modal attention interaction mechanism to obtain a first hybrid feature guided by vision and a second hybrid feature guided by audio; The comprehensive visual features and the auditory features are respectively used as inputs of the self-attention interaction mechanism, and are processed by the self-attention interaction mechanism to obtain single visual features and single audio features.

5. The audio and video content analysis method according to claim 2, characterized in that: The step of performing modal perception on the sample feature information by the modal perception module to obtain a modal perception result includes: Inputting the first mixed feature and the second mixed feature into the mixed modality fully connected layer respectively, and calculating a first mixed modality perceptual evidence value for the visual segment and a second mixed modality perceptual evidence value for the audio segment; Using the additive attention mechanism and the modality-aware fully connected layer, aggregating all of the first mixed modality perception evidence values ​​to obtain a visual mixed modality perception evidence value, and aggregating all of the second mixed modality perception evidence values ​​to obtain an audio mixed modality perception evidence value; Inputting the single visual feature and the single audio feature into the mixed modality fully connected layer respectively, and calculating the mixed modality perceptual evidence value of the visual segment and the mixed modality perceptual evidence value of the audio segment; Using the additive attention mechanism and the modality-aware fully connected layer, aggregating the mixed-modality perception evidence values ​​of all the visual segments to obtain video unimodal perception evidence values ​​and a visual attention aggregation matrix, and aggregating the mixed-modality perception evidence values ​​of all the audio segments to obtain audio unimodal perception evidence values ​​and an audio attention aggregation matrix; Based on the mixed modality perception evidence value of the visual segment and the mixed modality perception evidence value of the audio segment, the probability value of the auditory modality containing each category of action event and the probability value of the visual modality containing each category of action event are calculated.

6. The audio and video content analysis method according to claim 2, characterized in that: The step of perceiving the first mixed feature and the second mixed feature by the time series perception module to obtain a time series perception result includes: The first mixed feature and the second mixed feature are perceived by the foreground perception fully connected layer, the audio mapping fully connected layer, and the visual mapping fully connected layer to obtain a foreground segment perception evidence value; The first mixed feature and the second mixed feature are perceived by the background perception fully connected layer, the audio mapping fully connected layer, and the visual mapping fully connected layer to obtain a background segment perception evidence value; Adopting the additive attention mechanism and the temporal perception fully connected layer, respectively aggregating the foreground segment perception evidence value and the background segment perception evidence value to obtain a temporal attention aggregation matrix, a foreground temporal perception evidence value, and a background temporal perception evidence value; According to the foreground temporal perception evidence value, the probability value of the action event containing each category is calculated.

7. The audio and video content analysis method according to claim 2, characterized in that: The step of performing attention mechanism alignment according to the audio attention aggregation matrix, the visual attention aggregation matrix, and the temporal perception evidence value, and calculating the foreground and background consistency loss value, includes: Using a preset consistency loss function, the audio attention aggregation matrix, the visual attention aggregation matrix, and the temporal perception evidence value are aligned by an attention mechanism to obtain a foreground and background consistency loss value; The consistency loss function includes: in, Characterizes the consistency loss value of the foreground and background, The total number of categories representing action events, Representing the temporal attention aggregation matrix, Characterize the audio attention aggregation matrix, Characterizing visual attention aggregation matrices.

8. The audio and video content analysis method according to claim 1, characterized in that: The step of splitting and extracting features from the audio and video to be analyzed to obtain feature information includes: Splitting the audio and video to be parsed into audio to be parsed and video to be parsed, splitting the audio to be parsed into multiple audio segments, and splitting the video to be parsed into multiple video segments; Using a pre-trained 2D convolutional neural network and a pre-trained 3D convolutional neural network, respectively extracting features from each of the video clips to obtain 2D visual features and 3D visual features corresponding to each of the video clips; A pre-trained VGGish network is used to extract features from each audio clip to obtain auditory features corresponding to each audio clip.

9. An audio and video content analysis device, characterized in that: Used to implement the audio and video content analysis method according to any one of claims 1 to 8, comprising a feature extraction module, a feature fusion module, an input module and a parsing module; The feature extraction module is used to obtain the audio and video to be analyzed, split the audio and video to be analyzed, and extract features to obtain feature information; wherein the feature information includes 2D visual features, 3D visual features, and auditory features; The feature fusion module is used to fuse the 2D visual features and the 3D visual features to obtain a comprehensive visual feature; The input module is used to input the auditory features and the comprehensive visual features into a pre-optimized parsing algorithm; wherein the parsing algorithm is an parsing algorithm obtained by optimization training using weakly supervised learning; The parsing module is used to model and predict the modality of the auditory features and the comprehensive visual features through the parsing algorithm to obtain parsing results; wherein the parsing results include the action events contained in the audio and video to be parsed and the category and modality to which each action event belongs, and the modalities include vision and hearing.

Images