Audiovisual event identification and positioning method, system and device based on semantic consistent fragment selection and medium

Abstract

The invention discloses an audiovisual event identification and positioning method, system and device based on semantic consistency fragment selection and a medium. The method comprises the following steps: constructing an audiovisual event identification and positioning model based on semantic consistency fragment selection; designing a loss function, continuously training and optimizing the audiovisual event recognition and positioning model based on semantic consistent fragment selection through the loss function, and when the loss function is minimum, obtaining an optimal audiovisual event recognition and positioning model based on semantic consistent fragment selection; inputting the target video into the optimal audio-visual event recognition and positioning model based on semantic consistent fragment selection to obtain optimal target event recognition accuracy and positioning information of the target event; the system, the equipment and the medium are used for implementing the method. The invention provides a multi-modal similarity learning model and a global semantic perception and enhancement module to solve the problem of semantic imbalance between audio and video clips and improve the semantic consistency of audio-visual modalities.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio-visual event positioning, and in particular relates to an audio-visual event recognition and positioning method, system, device and medium based on semantically consistent segment selection. Background Art

[0002] In recent years, with the expanding application of deep learning, audio-visual learning has also attracted considerable attention. Audio-visual learning aims to mimic the way humans perceive the environment through both auditory and visual perception, leveraging the relationship between audio and visual modalities to address challenging machine perception problems. Audio-visual event recognition and localization is a key task in audio-visual learning. Its goal is to perceive and locate targets in a visual environment through the audio-visual modality, overcoming the limitations of perceptual localization tasks based on a single modality.

[0003] Visual and auditory information are important sources of information for humans to perceive their surroundings. As the two most widely studied organs, vision and hearing play a crucial role in the human nervous system. The human brain perceives the surrounding environment by integrating multimodal visual and auditory information. Therefore, the importance of audiovisual modalities in human perceptual localization provides a cognitive foundation for audiovisual event recognition and localization tasks. The interaction and integration of audiovisual modalities can serve as an important basis for promoting audiovisual learning.

[0004] For example, in real life, we can determine a dog's exact location by its appearance or its barking. However, when the dog's appearance is difficult to observe or its barking is very faint, it is difficult to determine the dog's location. However, we can simultaneously determine the dog's location by using both its barking and its appearance (semantic consistency) to determine its exact location. Therefore, semantic consistency of audiovisual modalities is an important foundation for audiovisual learning and a key development direction in audiovisual event recognition and localization tasks.

[0005] Current research on audiovisual event recognition and localization primarily encompasses audiovisual enhancement, cross-modal perception, and audiovisual collaboration. Similar to how people associate sounds with images and images with corresponding ones, this information consistency between visual and auditory modalities provides the foundation for cross-modal perception. Similar to the intermodal information interaction within the human brain's cortical regions, exploring effective audiovisual collaboration can achieve deeper semantic understanding.

[0006] In order to improve the accuracy of audiovisual event recognition and localization tasks, people have explored and solved many problems, but existing methods still have some shortcomings:

[0007] 1. To better utilize the consistency information of the audiovisual modalities, existing methods improve the semantic consistency of the audiovisual modalities by modeling the similarity between visual and auditory segments and using thresholding to filter out semantically dissimilar segments. However, existing methods filter out many similar segments in the process of propagating similar semantics, resulting in a loss of segment information in the audiovisual modalities.

[0008] 2. Existing methods focus on dividing a complete video into several segments and identifying and localizing the target by comparing the semantic consistency of the target in the auditory and visual segments. However, this approach ignores the influence of the global semantics of the complete video on the segment-level semantics.

[0009] Patent application publication number CN 117037046 A discloses an audiovisual event detection method, apparatus, storage medium, and electronic device. This invention discloses an audiovisual event detection method that extracts target video and audio from target audio and video data, segments the target video and audio, and uses the fusion of video and audio features from the same event segment to represent the audiovisual event semantics of the audio and video pair, ultimately determining the audiovisual event detection result. This invention effectively integrates audio and video features, but because the target video and audio are segmented, it ignores the semantic consistency between segment-level video and segment-level audio, and also ignores the correlation between adjacent segment pairs, which can lead to a loss of audiovisual semantics. Summary of the Invention

[0010] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an audio-visual event recognition and positioning method, system, device and medium based on semantically consistent segment selection, and propose a multimodal similarity learning model and a global semantic perception and enhancement module to solve the problem of semantic imbalance between audio and video segments and improve the semantic consistency of audio-visual modalities; the present invention constructs a multimodal similarity learning model, calculates the similarity coefficient matrix of the audio-visual modality and the similarity coefficient matrix of the multimodal, and introduces a random feature mask, which can perform fine-grained encoding of the semantic relationship of the audio-visual modality, and has the effect of learning the semantic consistency of the audio-visual modalities more closely; at the same time, by introducing cross-modal video-level semantic features through the global semantic perception and enhancement module, it can enhance the segment-level semantics by introducing prior knowledge (cross-modal video-level semantics), which has the effect of enhancing the semantic consistency of the audio-visual modalities.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] A method for audiovisual event recognition and location based on semantically consistent segment selection includes the following steps:

[0013] Step 1: Build an audiovisual event recognition and localization model based on semantically consistent segment selection;

[0014] Step 2: Design a loss function. Use this loss function to continuously train and optimize the audiovisual event recognition and localization model based on semantically consistent segment selection constructed in Step 1. When the loss function is minimized, the optimal audiovisual event recognition and localization model based on semantically consistent segment selection is obtained.

[0015] Step 3: Input the target video into the optimal audiovisual event recognition and localization model based on semantically consistent segment selection obtained in step 2 to obtain the optimal target event recognition accuracy and target event localization information.

[0016] The specific method of step 1 includes:

[0017] Step 1.1: Obtain the target video, preprocess the target video to obtain the video signal and audio signal, and use the convolutional neural network to extract the visual features of the video signal and the auditory features of the audio signal;

[0018] Step 1.2: The visual features of the video signal and the auditory features of the audio signal extracted in step 1.1 are input into the attention module to obtain visual features guided by the auditory features. The attention module uses the query vector of the auditory features of the audio signal to extract visual features related to the auditory features, and uses the auditory features to focus on the area of interest in the visual features.

[0019] The attention module is expressed as follows:

[0020] w t =Softmax(x t )

[0021] x t =W f σ(W v U v (v t )+(W a U a (a t )))1 T

[0022] Among them, U v and U a are two transformation functions that project video features and audio features to the same dimension d, W v , W a , W f represents the learnable parameters, 1 T The terms in are all 1, σ is the hyperbolic tangent function, w t is the calculated attention map;

[0023] Step 1.3: Input the auditory features of the audio signal extracted in step 1.1 and the visual features guided by the auditory features obtained in step 1.2 into the bidirectional long short-term memory network (Bi-LSTM) for spatiotemporal feature extraction and modeling, and obtain the auditory feature a with spatiotemporal information. lstm and visual features v lstm ;

[0024] Step 1.4: Construct a multimodal similarity learning model and transform the auditory feature a with spatiotemporal information obtained in step 1.3 into lstm and visual features v lstm The input is into the multimodal similarity learning model, which uses the similarity graph between the visual segment features and the auditory segment features guided by the auditory features, the similarity graph of the intersection of the visual segment features and the auditory segment features guided by the auditory features, and the similarity graph of the combined features, and introduces feature random masks to generate similarity scores of segment features, model the semantic similarity of the audiovisual segments, select segments with high similarity scores, obtain segments with highly consistent semantics, and obtain visual features v with highly consistent semantics of the segments. MSLM and auditory features a MSLM ;

[0025] Step 1.5: Build a global semantic perception and enhancement module;

[0026] The global semantic perception and enhancement module consists of attention-based relation perception (Relation Aware), cross-relation perception (Audio-Video Interaction), and a bidirectional gated recurrent neural network (Bi-GRU). The global semantic perception and enhancement module perceives the video-level global semantics of a complete video in the temporal dimension through attention-based relation perception (Relation Aware) and cross-relation perception (Audio-Video Interaction);

[0027] Attention-based relation awareness is expressed as:

[0028] F out =Softmax(QK T )V,

[0029] Q=FW Q ,K=FW K ,V=FW V ,

[0030] Based on cross-relationship perception (Audio-Video Interaction) it is expressed as:

[0031] F va =Fv ⊙F a

[0032] F c =concat(F v ,F a )

[0033]

[0034] q1=vW Q ,K 1,2 =m a,v W K ,V 1,2 =F a,v W V

[0035] Among them, Q, K, V represent query, key, and value, and W Q ,W K ,W V represents the learnable parameter matrix, F represents the input features of attention-based relational perception, and F out represents the output features of attention-based relation perception, F v , F a represents the input features based on cross-relation perception, F av =AVI(v,a) represents the output features of cross-relation perception;

[0036] The visual features v with highly consistent semantics of the fragments obtained in step 1.4 are MSLM and auditory features a MSLM Input into the global semantic perception and enhancement module, the visual features v with highly consistent semantics of the fragments MSLM and auditory features a MSLM First, it is transposed and input into the attention-based relational perception. The result is transposed again and input into the cross-relational perception, and the cross feature F is output. av ;

[0037] Cross feature F av The video-level global semantics of the video is expressed, and the video-level global semantics of the video is introduced as a priori knowledge into the bidirectional gated recurrent neural network (Bi-GRU). The bidirectional gated recurrent neural network (Bi-GRU) transforms the cross feature F av As the common initial hidden state of two independent gated recurrent neural networks (GRU) to guide the dynamic modeling of visual and auditory features, and enhance the visual features v with high semantic consistency of the segment obtained in step 1.4 MSLM and auditory features a MSLM The semantics of the enhanced fragment is obtained by highly consistent visual features V ISCEand auditory feature A ISCE ; This process means:

[0038] A ISCE =GRU(A seg ,F av ),

[0039] V ISCE =GRU(V seg ,F av ),

[0040] Step 1.6: The enhanced visual features V with highly consistent semantics of the fragments obtained in step 1.5 are ISCE and auditory feature A ISCE The fused features are input into the classification and recognition module to identify and locate the event categories in the target video.

[0041] The implementation method of step 1.1 includes:

[0042] Step 1.11: Get the target video The target video is decomposed into T continuous segments of 1 second in length. Each segment consists of a synchronized video signal segment and an audio signal segment. t V , S t A Represented as video signal segments and audio signal segments respectively;

[0043] Step 1.12: Sample each video signal segment at a fixed number of frames per second to obtain a fixed number of images for each video signal segment. Use a convolutional neural network (VGG-19) to extract feature maps of the fixed number of images for each video signal segment. Globally merge the feature maps of the fixed number of images per second to obtain a global feature map, which is the segment-level feature of each video signal segment.

[0044] Step 1.13: Convert each audio signal segment into a log-mel spectrogram and use a convolutional neural network (CNN) to extract segment-level features for each audio signal segment.

[0045] Step 1.14: Arrange the segment-level features of each video signal segment extracted in step 1.12 and the segment-level features of each audio signal segment extracted in step 1.13 in temporal order to form the visual features of the video signal and the auditory features of the audio signal.

[0046] The implementation method of step 1.4 includes:

[0047] Step 1.41: Encode intramodal and cross-modal relationships between audiovisual modalities

[0048] For the auditory feature a with spatiotemporal information obtained in step 1.3 lstm and visual features v lstm , firstly transform the auditory feature a lstm and visual features v lstm Connect to obtain the combined feature c lstm , and auditory feature a lstm , visual features v lstm and combined feature c lstm Performing linear transformation to obtain linearly transformed auditory segment features, visual segment features, and combined segment features;

[0049] Step 1.42: Multimodal Similarity Learning:

[0050] Transpose the auditory segment features after the linear transformation in step 1.41 and perform matrix multiplication with the visual segment features after the linear transformation to obtain the similarity map β between the visual segment features after the linear transformation and the auditory segment features. va ; The similarity graph β between the linearly transformed visual segment features and the auditory segment features va Perform the transposition operation to obtain the similarity graph β between the linearly transformed auditory segment features and visual segment features av , perform a transposition operation on the combined segment features after the linear transformation in step 1.41, and perform a matrix multiplication operation on the combined segment features after the linear transformation to obtain a similarity graph of the combined segment features; and perform a feature random masking operation on the similarity graph of the combined segment features, with a masking ratio of 0%-100%, to obtain a masked similarity graph of the combined segment features;

[0051] The process of step 1.41 and step 1.42 is expressed by the following formula:

[0052]

[0053] Among them, β va , β av , β c , They represent the similarity graph between the visual segment features and the auditory segment features after linear transformation, the similarity graph between the auditory segment features and the visual segment features after linear transformation, the similarity graph of the combined segment features and the similarity graph of the combined segment features after masking, W1 v , W1 a , W1 c , represents the linear transformation operation, (·) T represents the transpose operation, d l Represents feature dimension;

[0054] Step 1.43: Filtering of audiovisual modality consistent semantic segments:

[0055] The thresholding method is used to obtain the similarity graph β between the linearly transformed visual segment features and the auditory segment features obtained in step 1.42. va , the similarity graph β between the auditory segment features and the visual segment features after linear transformation av The visual segment features and auditory segment features with low similarity are connected and filtered, and only the visual segment features and auditory segment features with high semantic consistency are retained to obtain the similarity graph γ between the visual segment features and auditory segment features with high similarity. va Similarity graph between auditory segment features and visual segment features with high similarity γ av ;

[0056] The above process is expressed as:

[0057] γ va =β va Γ(β va -τ),

[0058] γ av =β av Γ(β av -τ),

[0059] Where τ represents a trainable hyperparameter and Γ(·) represents a thresholding operation that returns 1 if the input is greater than or equal to zero, otherwise it returns 0.

[0060] Step 1.44: Perform matrix multiplication or dot product operations on different similarity graphs and different features to obtain the aggregation of audio-visual features;

[0061] The specific method includes: lstm Similarity graph of combined fragment features after masking obtained in step 1.42 Perform a dot product operation and separate the results through the Split operation to obtain the auditory calibration feature a co and visual calibration features v co ; Compare the similarity graph γ between the linearly transformed visual segment features obtained in step 1.41 and the highly similar auditory segment features and visual segment features obtained in step 1.43 av Perform matrix multiplication to obtain high-similarity visual features; Compare the linearly transformed auditory segment features obtained in step 1.41 with the high-similarity visual segment features and auditory segment features obtained in step 1.43 to form a similarity graph γ. va Perform matrix multiplication to obtain auditory features with high similarity; combine the visual features with high similarity with the visual calibration features v co Perform a dot product operation to combine the auditory features with high similarity with the auditory calibration features a coPerform a dot product operation and combine the two results with the auditory feature a with spatiotemporal information obtained in step 1.3 lstm and visual features v lstm Perform cross-addition to obtain visual features v with highly consistent semantics of the fragments MSLM and auditory features a MSLM ;

[0062] The above process is expressed as:

[0063] v co ,a co =Split(β c Fc lstm ),

[0064]

[0065] Among them, F represents the fully connected layer;

[0066] The workflow of step 1.6 includes:

[0067] Step 1.61: The enhanced visual features V with highly consistent semantics of the fragment obtained in step 1.5 ISCE and auditory feature A ISCE Obtain fusion features by adding features;

[0068] Step 1.62: Input the fused features obtained in step 1.61 into the classification and recognition module. The classification and recognition module includes two fully connected layers. The audio-visual event recognition is performed based on the fused features to obtain the identification and location information of the target event.

[0069] In step 2, the loss function is calculated as follows: according to different audiovisual event localization tasks, the loss function is divided into a loss function under full supervision and a loss function under weak supervision;

[0070] The loss function in the fully supervised setting is composed of the event-related score s r , known segment-level event label Y tc , background label y t ={y t ||y t ∈{0,1},t=1,...,T} and the segment-level event labels O predicted by the model in the fully supervised setting tc Calculation, the calculation process is:

[0071] AV=W f (A ISCE +V ISCE ),

[0072] s r =δ(AV·F1),

[0073] s=A ISCE ⊙V ISCE / ||A ISCE ⊙V ISCE ||1

[0074]

[0075] Where F1 represents the fully connected layer, AV represents the visual features with spatiotemporal information output by the Bi-LSTM (Bi-LSTM) and the audio features with spatiotemporal information are added together to obtain the fusion features, δ represents the activation function, and s represents the similarity vector of the segment-level visual and audio features; y tl represents y t The regularization result, L BCE (s r ,y t ) represents the event-related score s r and background label y t ={y t ||y t ∈{0,1},t=1,...,T} cross entropy loss, L MSE (s,y tl ) represents the similarity vectors s and y of segment-level visual and audio features t The regularization result y tl The mean square error of

[0076] The loss function in the weakly supervised setting is the segment-level event labels predicted by the model in the weakly supervised setting. and video-level labels Y in a weakly supervised setting c Calculation, the calculation process is:

[0077]

[0078] Among them, s represents the softmax function and λ represents the hyperparameter.

[0079] The present invention also provides an audiovisual event recognition and positioning system based on semantically consistent segment selection, comprising:

[0080] A model building module, used to build an audiovisual event recognition and localization model based on semantically consistent segment selection;

[0081] The model training module is used to implement the designed loss function, and continuously train and optimize the audiovisual event recognition and localization model based on semantically consistent segment selection through the loss function. When the loss function is minimized, the optimal audiovisual event recognition and localization model based on semantically consistent segment selection is obtained;

[0082] The recognition and positioning result output module is used to input the target video into the optimal audio-visual event recognition and positioning model based on semantically consistent segment selection to obtain the optimal target event recognition accuracy and target event positioning information.

[0083] The present invention also provides an audiovisual event recognition and positioning device based on semantically consistent segment selection, comprising:

[0084] Memory: a computer-readable device storing a computer program for the above-mentioned method for identifying and locating audiovisual events based on selection of semantically consistent segments;

[0085] Processor: used to implement the audiovisual event recognition and positioning method based on semantically consistent segment selection when executing the computer program.

[0086] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the audiovisual event recognition and positioning method based on semantically consistent segment selection.

[0087] Compared with the prior art, the present invention has the following beneficial effects:

[0088] 1. The present invention designs a multimodal similarity learning module, models the similarity graphs between visual segment features and auditory segment features, and the similarity graphs between auditory segment features and visual segment features, filters out pairs of visual segments and auditory segments with low similarity, and selects pairs of visual segments and auditory segments with high semantic consistency for forward propagation, thereby enhancing the semantic consistency of audiovisual modalities.

[0089] 2. In the multimodal similarity learning module, the present invention combines visual and auditory features to obtain combined features and a similarity graph of combined segment features, and then performs a random feature masking operation on the similarity graph of combined segment features. Through masking and audiovisual feature aggregation, some feature information is retained, thereby smoothing the data and enhancing the robustness of the multimodal similarity learning model. This reduces the loss of segment information during the selection of visual and auditory segments with high semantic consistency, thereby enhancing the semantic consistency of the audiovisual segments and improving recognition and positioning accuracy.

[0090] 3. The present invention designs a global semantic perception and enhancement module to perceive video-level features in the time dimension of the complete video, uses video-level features as prior knowledge, and uses a bidirectional gated recurrent neural network (Bi-GRU) to focus on the differential semantics between video-level features and segment-level features. These differential semantics are used to enhance the semantic consistency of the audiovisual modality, thereby improving the semantic consistency of the audiovisual modality.

[0091] In summary, the present invention first uses convolutional neural networks (CNN) and convolutional neural networks (VGG) networks to extract the auditory signal features and visual signal features of the video, adds an attention module, uses the query vector of the auditory features to extract the visual features related to the auditory features, uses Bi-LSTM to model the auditory features and uses the query vector of the auditory features to extract the temporal characteristics of the visual features related to the auditory features, then performs multimodal similarity learning modeling on the auditory results and the visual results, learns the similarity graph between the visual segment features and the auditory segment features and the similarity graph of the combined segment features, uses the thresholding method to filter out segments with inconsistent audiovisual semantics, improves the semantic consistency of the audiovisual modality, and introduces a feature random mask operation on the similarity graph to reduce the loss of segment information in the filtering process. Finally, the present invention designs a global semantic perception and enhancement module to model the global video-level semantics of the complete video, and uses the difference semantics between the global video-level semantics and the segment-level semantics to enhance the segment-level semantics, thereby improving the semantic consistency of the audiovisual modality and improving the accuracy of audiovisual event recognition and positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 It is a schematic diagram of the overall network structure of the present invention.

[0093] Figure 2 This is a visualization diagram of the test results of the present invention; it includes a heat map of the target position of the present invention and other methods. DETAILED DESCRIPTION

[0094] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings.

[0095] The present invention first uses convolutional neural networks (CNNs) and convolutional neural networks (VGGs) to extract the auditory signal features and visual signal features of the video, respectively. Based on the attention mechanism, the query vector of the auditory features is used to extract visual features related to the auditory features, and the visual features associated with the auditory features are learned. A bidirectional long short-term memory network (Bi-LSTM) is used to model the temporal characteristics of the auditory features and the visual features associated with the auditory features. The results are then subjected to multimodal similarity learning modeling to learn similarity graphs of visual segment features and auditory segment features, as well as a similarity graph of combined segment features. The similarity graphs are thresholded and subjected to feature random masking operations to filter out semantically inconsistent visual and auditory segment features, improve the semantic consistency of the audiovisual modalities, and reduce the loss of segment information during the filtering process. The present invention designs a global semantic perception and enhancement module. This module uses a feature sensor to perceive the global video-level semantics of the complete video, combines a bidirectional gated recurrent neural network (Bi-GRU) to focus on the difference semantics between the global video-level semantics and the segment-level semantics, and uses the difference semantics to enhance the segment-level semantics, thereby improving the semantic consistency of the audiovisual modalities. Through multimodal similarity learning and global semantic perception and enhancement modules, segments with high semantic consistency are selected to improve the accuracy of audiovisual event recognition and positioning.

[0096] like Figure 1 As shown, a method for audiovisual event recognition and localization based on semantically consistent segment selection includes the following steps:

[0097] Step 1: Build an audiovisual event recognition and localization model based on semantically consistent segment selection;

[0098] Step 1.1: Obtain the target video, preprocess the target video to obtain the video signal and audio signal, and use the convolutional neural network (VGG) and convolutional neural network (CNN) to extract the visual features of the video signal and the auditory features of the audio signal respectively;

[0099] The specific implementation steps of step 1.1 include:

[0100] Step 1.11: Get the target video The target video is decomposed into T continuous segments of 1 second in length. Each segment consists of a synchronized video signal segment and an audio signal segment. t V , S t A Represented as video signal segments and audio signal segments respectively;

[0101] Step 1.12: Obtain 16 frames of images for each video signal segment at a sampling frequency of 16 frames per second. Use a convolutional neural network (VGG-19) to extract feature maps of the 16 frames of images for each video signal segment. Combine the 16 frame feature maps per second through global merging to obtain segment-level features for each video signal segment.

[0102] Step 1.13: Sample each audio signal segment at 16000 Hz and convert it into a log-mel spectrogram. Use a convolutional neural network (CNN) to extract segment-level features for each audio signal segment.

[0103] Step 1.14: Arrange the segment-level features of each video signal segment extracted in step 1.12 and the segment-level features of each audio signal segment extracted in step 1.13 in temporal order to form visual features of the video signal and auditory features of the audio signal.

[0104] Step 1.2: The visual features of the video signal and the auditory features of the audio signal extracted in step 1.1 are input into the attention module to obtain visual features guided by the auditory features. The attention module uses the query vector of the auditory features of the audio signal to extract visual features related to the auditory features, and uses the auditory features to focus on the area of interest in the visual features.

[0105] The attention module is expressed as follows:

[0106] w t =Softmax(x t )

[0107] x t =W f σ(W v U v (v t )+(W a U a (a t )))1 T

[0108] Among them, U v and U a are two transformation functions that project video features and audio features to the same dimension d, W v , W a , W f represents the learnable parameters, 1 T The terms in are all 1, σ is the hyperbolic tangent function, w t is the calculated attention map;

[0109] Step 1.3: Input the auditory features of the audio signal extracted in step 1.1 and the visual features guided by the auditory features obtained in step 1.2 into the bidirectional long short-term memory network (Bi-LSTM) for spatiotemporal feature extraction and modeling, and obtain the auditory feature a with spatiotemporal information. lstm and visual features v lstm ;

[0110] Step 1.4: Construct a multimodal similarity learning model and transform the auditory feature a with spatiotemporal information obtained in step 1.3 into lstm and visual features v lstm Input into the multimodal similarity learning model to forward propagate audiovisual clips with high semantic consistency. The multimodal similarity learning model uses the similarity graph between the visual clip features and the auditory clip features guided by auditory features, the similarity graph of the intersection of the visual clip features and the auditory clip features guided by auditory features, and the similarity graph of the combined features, and introduces feature random masks to generate similarity scores of clip features, model the semantic similarity of audiovisual clips, and select clips with high similarity scores. Segments with high semantic consistency are obtained, and visual features v with high semantic consistency of the segments are obtained. MSLM and auditory features a MSLM In this way, audiovisual clips with high semantic consistency are forward propagated, the semantic consistency of the audiovisual modalities is improved, and the semantic information of the visual and auditory modalities is aligned.

[0111] The implementation method of step 1.4 includes:

[0112] Step 1.41: Encode intramodal and cross-modal relationships between audiovisual modalities

[0113] For the auditory feature a with spatiotemporal information obtained in step 1.3 lstm and visual features v lstm , firstly transform the auditory feature a lstm and visual features v lstm Connect to obtain the combined feature c lstm , and auditory feature a lstm , visual features v lstm and combined feature c lstm Performing linear transformation to obtain linearly transformed auditory segment features, visual segment features, and combined segment features;

[0114] Step 1.42: Multimodal Similarity Learning:

[0115] Transpose the auditory segment features after the linear transformation in step 1.41 and perform matrix multiplication with the visual segment features after the linear transformation to obtain the similarity map β between the visual segment features after the linear transformation and the auditory segment features. va; The similarity graph β between the linearly transformed visual segment features and the auditory segment features va Perform the transposition operation to obtain the similarity graph β between the linearly transformed auditory segment features and visual segment features av , perform a transposition operation on the combined segment features after the linear transformation in step 1.41, and perform a matrix multiplication operation on the combined segment features after the linear transformation to obtain a similarity graph of the combined segment features; and perform a feature random masking operation on the similarity graph of the combined segment features with a masking ratio of %25 to obtain a masked similarity graph of the combined segment features;

[0116] The process of step 1.41 and step 1.42 is expressed as follows:

[0117]

[0118] Among them, β va , β av , β c , They represent the similarity graph between the visual segment features and the auditory segment features after linear transformation, the similarity graph between the auditory segment features and the visual segment features after linear transformation, the similarity graph of the combined segment features and the similarity graph of the combined segment features after masking, W1 v , W1 a , W1 c , represents the linear transformation operation, (·) T represents the transpose operation, d l Represents feature dimension;

[0119] Step 1.43: Filtering of audiovisual modality-consistent semantic segments.

[0120] The thresholding method is used to obtain the similarity graph β between the linearly transformed visual segment features and the auditory segment features obtained in step 1.42. va , the similarity graph β between the auditory segment features and the visual segment features after linear transformation av The visual segment features and auditory segment features with low similarity are connected and filtered, and only the visual segment features and auditory segment features with high semantic consistency are retained to obtain the similarity graph γ between the visual segment features and auditory segment features with high similarity. va Similarity graph between auditory segment features and visual segment features with high similarity γ av ;

[0121] The above process is expressed as:

[0122] γ va =β va Γ(β va -τ),

[0123] γ av =β av Γ(β av -τ),

[0124] Where τ represents a trainable hyperparameter and Γ(·) represents a thresholding operation that returns 1 if the input is greater than or equal to zero, otherwise it returns 0.

[0125] Step 1.44: Perform matrix multiplication or dot product operations on different similarity graphs and different features to obtain the aggregation of audio-visual features. Through this process, the forward propagation of semantic segments consistent with the audio-visual modality is achieved.

[0126] The specific method includes: lstm Similarity graph of combined fragment features after masking obtained in step 1.42 Perform a dot product operation and separate the results through the Split operation to obtain the auditory calibration feature a co and visual calibration features v co ; Compare the similarity graph γ between the linearly transformed visual segment features obtained in step 1.41 and the highly similar auditory segment features and visual segment features obtained in step 1.43 av Perform matrix multiplication to obtain high-similarity visual features; Compare the linearly transformed auditory segment features obtained in step 1.41 with the high-similarity visual segment features and auditory segment features obtained in step 1.43 to form a similarity graph γ. va Perform matrix multiplication to obtain auditory features with high similarity; combine the visual features with high similarity with the visual calibration features v co Perform a dot product operation to combine the auditory features with high similarity with the auditory calibration features a co Perform a dot product operation and combine the two results with the auditory feature a with spatiotemporal information obtained in step 1.3 lstm and visual features v lstm Perform cross-addition to obtain visual features v with highly consistent semantics of the fragments MSLM and auditory features a MSLM ; Through this process, the forward propagation of semantic segments consistent with the audio-visual modality is achieved.

[0127] The above process can be expressed as:

[0128] v co ,a co =Split(β c Fc lstm ),

[0129]

[0130] Among them, F represents the fully connected layer;

[0131] Step 1.5: Build a global semantic perception and enhancement module;

[0132] The global semantic perception and enhancement module consists of attention-based relation perception (Relation Aware), cross-relation perception (Audio-Video Interaction), and a bidirectional gated recurrent neural network (Bi-GRU). The attention-based relation perception (Relation Aware) and cross-relation perception (Audio-Video Interaction) perceive the video-level global semantics of a complete target video in the temporal dimension. The global semantic perception and enhancement module uses attention-based relation perception (Relation Aware) and cross-relation perception (Audio-Video Interaction) to perceive the video-level global semantics of a complete target video in the temporal dimension.

[0133] Attention-based relation awareness is expressed as:

[0134] F out =Softmax(QK T )V,

[0135] Q=FW Q ,K=FW K ,V=FW V ,

[0136] Based on cross-relationship perception (Audio-Video Interaction) it is expressed as:

[0137] F va =F v ⊙F a

[0138] F c =concat(F v ,F a )

[0139]

[0140] q1=vW Q ,K 1,2 =m a,v W K ,V 1,2 =F a,v W V

[0141] Among them, Q, K, V represent query, key, and value, and W Q ,W K,W V represents the learnable parameter matrix, F represents the input features of attention-based relational perception, and F out represents the output features of attention-based relation perception, F v , F a represents the input features based on cross-relation perception, F av =AVI(v,a) represents the output features of cross-relation perception;

[0142] The visual features v with highly consistent semantics of the fragments obtained in step 1.4 are MSLM and auditory features a MSLM Input into the global semantic perception and enhancement module, the visual features v with highly consistent semantics of the fragments MSLM and auditory features a MSLM First, it is transposed and input into the attention-based relational perception. The result is transposed again and input into the cross-relational perception, and the cross feature F is output. av ;

[0143] Cross feature F av The video-level global semantics of the video is expressed, and the video-level global semantics of the video is introduced as a priori knowledge into the bidirectional gated recurrent neural network (Bi-GRU). The bidirectional gated recurrent neural network (Bi-GRU) transforms the cross feature F av As the common initial hidden state of two independent gated recurrent neural networks (GRUs) to guide the dynamic modeling of visual features and auditory features, in this way, the gated recurrent neural network (GRU) can focus on important features that are consistent with the semantics of video-level events, rather than treating every feature indiscriminately at the beginning. In addition, the cross feature F av Including visual and auditory semantics of events, cross features F av It can be used as supplementary information to help the model learn more powerful semantic continuous representation, model the temporal changes of visual (auditory) features, and enhance the visual features v with highly consistent semantics of the segments obtained in step 1.4. MSLM and auditory features a MSLM The semantics of the enhanced fragment is obtained by highly consistent visual features V ISCE and auditory feature A ISCE ; This process means:

[0144] A ISCE =GRU(A seg ,F av ),

[0145] V ISCE =GRU(V seg ,F av ),

[0146] Through this step, GRU can focus on the differential semantics carried by video-level feature representations, while audio (visual) features can utilize these differential semantics to enhance semantically consistent representations.

[0147] Step 1.6: The enhanced visual features V with highly consistent semantics of the fragments obtained in step 1.5 are ISCE and auditory feature A ISCE The fused features are obtained by adding features, and the fused features are input into the classification and recognition module. The classification and recognition module includes two fully connected layers to realize the recognition and positioning of event categories in the target video.

[0148] The workflow includes:

[0149] Step 1.61: The enhanced visual features V with highly consistent semantics of the fragment obtained in step 1.5 ISCE and auditory feature A ISCE Obtain fusion features by adding features;

[0150] Step 1.62: Input the fused features obtained in step 1.61 into a classification and recognition module consisting of two fully connected layers, and perform audio-visual event recognition based on the fused features to obtain identification and location information of the target event.

[0151] Step 2: Design a loss function, and continuously train and optimize the audio-visual event recognition and localization model based on semantically consistent segment selection constructed in step 1 through the loss function. When the loss function is minimized, the optimal audio-visual event recognition and localization model based on semantically consistent segment selection is obtained; input the target video into the optimal audio-visual event recognition and localization model based on semantically consistent segment selection to obtain the optimal target event recognition accuracy and target event localization information.

[0152] The loss function is calculated as follows: according to different audiovisual event localization tasks, the loss function is divided into a loss function under full supervision and a loss function under weak supervision;

[0153] The loss function in the fully supervised setting is composed of the event-related score s r , known segment-level event label Y tc , background label y t ={y t ||y t ∈{0,1},t=1,...,T} and the segment-level event labels O predicted by the model in the fully supervised setting tc Calculation; the calculation process is:

[0154] AV=W f (A ISCE +V ISCE ),

[0155] s r =δ(AV·F1),

[0156] s=A ISCE ⊙V ISCE / ||A ISCE ⊙V ISCE ||1

[0157]

[0158] Where F1 represents the fully connected layer, AV represents the visual features with spatiotemporal information output by the Bi-LSTM (Bi-LSTM) and the audio features with spatiotemporal information are added together to obtain the fusion features, δ represents the activation function, and s represents the similarity vector of the segment-level visual and audio features; y tl represents y t The regularization result, L BCE (s r ,y t ) represents the event-related score s r and background label y t ={y t ||y t ∈{0,1},t=1,...,T} cross entropy loss, L MSE (s,y tl ) represents the similarity vectors s and y of segment-level visual and audio features t The regularization result y tl The mean square error of

[0159] The loss function in the weakly supervised setting is the segment-level event labels predicted by the model in the weakly supervised setting. and video-level labels Y in a weakly supervised setting c Calculation, the calculation process is:

[0160]

[0161] Among them, s represents the softmax function and λ represents the hyperparameter.

[0162] The present invention also provides an audiovisual event recognition and positioning system based on semantically consistent segment selection, comprising:

[0163] A model building module, used to implement the construction of an audiovisual event recognition and localization model based on semantically consistent segment selection in step 1;

[0164] A model training module is used to implement the loss function designed in step 2. The loss function is used to continuously train and optimize the audiovisual event recognition and localization model based on semantically consistent segment selection constructed in step 1. When the loss function is minimized, the optimal audiovisual event recognition and localization model based on semantically consistent segment selection is obtained.

[0165] The recognition and positioning result output module is used to implement step 3 by inputting the target video into the optimal audio-visual event recognition and positioning model based on semantically consistent segment selection obtained in step 2, so as to obtain the optimal target event recognition accuracy and positioning information of the target event.

[0166] The present invention also provides an audiovisual event recognition and positioning device based on semantically consistent segment selection, comprising:

[0167] Memory: a computer-readable device storing a computer program for the above-mentioned method for identifying and locating audiovisual events based on selection of semantically consistent segments;

[0168] Processor: used to implement the audiovisual event recognition and positioning method based on semantically consistent segment selection when executing the computer program.

[0169] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the audiovisual event recognition and positioning method based on semantically consistent segment selection.

[0170] like Figure 1 As shown, the present invention consists of three parts. The first part is the input end, which includes data preprocessing and data feature and time information extraction; the second part is a multimodal similarity learning model of audio-visual modalities, which learns the similarity between audio-visual modalities and forward propagates the segments of semantic similarity of audio-visual segments to improve the semantic consistency of audio-visual modalities. And introduces multimodal similarity graphs and feature random masks to reduce information loss in the propagation process. The third part is global semantic perception and enhancement, which uses attention-based perception and cross-relationship perception to perceive the global video-level semantics of the video, and uses Bi-GRU to focus on the difference semantics between the global video-level semantics and the segment-level semantics of the audio-visual signal, and uses the difference semantics to enhance the segment-level semantics, improve the semantic consistency of the audio-visual segments, thereby selecting audio-visual segments with high semantic consistency and improving the accuracy of audio-visual event recognition and positioning.

[0171] The effect of the present invention is further described below in conjunction with simulation experiments:

[0172] 1. Simulation Experiment Conditions

[0173] The hardware platform of the simulation experiment of the present invention is: the processor is Intel i7-8750H, and the GPU is NUVIDA GeForce RTX 4060.

[0174] The software platform of the simulation experiment platform of the present invention is: Ubuntu20.04 operating system and PyCharm 2022, PyTorch 1.12.0, and CUDA11.2.

[0175] 2. Simulation steps

[0176] The simulation process of the present invention uses the public data set The Audio-Visual Event dataset to evaluate the method of the present invention. The video data in the data set is input into this embodiment. The video in the data set is first preprocessed, and the visual features and auditory features are extracted respectively using a convolutional neural network (VGG) and a convolutional neural network (CNN). The visual features and auditory features are input into the Bi-LSTM module, and then the output results of the Bi-LSTM module are subjected to multimodal similarity modeling, and the audio-visual segments with high semantic consistency are forward propagated. Then, they are input into global perception and enhancement, and the segment-level semantics are enhanced using differential semantics. Finally, it is input into the recognition and positioning module, and the model is continuously trained and optimized through the loss function to achieve final accurate recognition and positioning.

[0177] 3. Simulation content and results analysis

[0178] The simulation experiment of this paper uses the public dataset The Audio-Visual Event dataset, which contains 4143 video clips of 28 categories, covering various audio-visual activities such as human speech, car driving, airplane roar, animal sounds, etc. Each video clip lasts for 10 seconds and is annotated at the second level. For weak supervision tasks, 178 unannotated noise samples are also introduced. The event classification of each video clip is predicted, and the overall recognition and positioning accuracy of the two AVE tasks is used as the performance evaluation indicator. The results are shown in Table 1 and Table 2. Figure 2 shown.

[0179] The simulation effect of the present invention is further described below in conjunction with Table 1.

[0180] Table 1 is a comparison table of the accuracy of the present invention and the existing method in identifying the AVE data set.

[0181]

[0182] The test dataset is a publicly available dataset from AVE. As shown in Table 1, the proposed method improves recognition and localization accuracy by 7.2% and 9.2% under full and weak supervision, respectively. It also achieves improved recognition and localization accuracy compared to the CMAN, AVSDN, PSP, and VSGG methods, demonstrating the effectiveness of the proposed method.

[0183] The following combination Figure 2 The simulation effect of the present invention is further described.

[0184] Figure 2 The visualization results of the model proposed in the present invention and AVEL, PSP and VSGG are shown. Specifically, it is observed that in the case of cats and dogs, the events are characterized by visual features and background textures, but the intensity of the audio component is weak, and the areas recognized by AVEL, PSP and VSG are relatively scattered. For events where visual information cannot be fully utilized, such as cars under occlusion and dark conditions, if there are no sound features, the visual features cannot correctly guide the event to the target area. The experimental results show that the model proposed in the present invention enhances the semantic consistency of audiovisual modalities by selecting semantically consistent segments, more effectively utilizing visual and data features, and better focusing on areas that are more closely related to audiovisual information, thereby achieving final accurate recognition and positioning.

Claims

1. A method for audiovisual event recognition and localization based on semantically consistent segment selection, characterized in that: The following steps are involved: Step 1: Build an audiovisual event recognition and localization model based on semantically consistent segment selection; Step 2: Design a loss function. Use this loss function to continuously train and optimize the audiovisual event recognition and localization model based on semantically consistent segment selection constructed in Step 1. When the loss function is minimized, the optimal audiovisual event recognition and localization model based on semantically consistent segment selection is obtained. Step 3: Input the target video into the optimal audiovisual event recognition and localization model based on semantically consistent segment selection obtained in step 2 to obtain the optimal target event recognition accuracy and target event localization information.

2. The method for audiovisual event recognition and location based on semantically consistent segment selection according to claim 1, characterized in that: The specific method of step 1 includes: Step 1.1: Obtain the target video, preprocess the target video to obtain the video signal and audio signal, and use the convolutional neural network to extract the visual features of the video signal and the auditory features of the audio signal; Step 1.2: The visual features of the video signal and the auditory features of the audio signal extracted in step 1.1 are input into the attention module to obtain visual features guided by the auditory features. The attention module uses the query vector of the auditory features of the audio signal to extract visual features related to the auditory features, and uses the auditory features to focus on the area of interest in the visual features. The attention module is expressed as follows: w t =Softmax(x t ) x t =W f σ(W v U v (v t )+(W a U a (a t )))1 T Among them, U v and U a are two transformation functions that project video features and audio features to the same dimension d, W v , W a , W f represents the learnable parameters, 1 T The terms in are all 1, σ is the hyperbolic tangent function, w t is the calculated attention map; Step 1.3: Input the auditory features of the audio signal extracted in step 1.1 and the visual features guided by the auditory features obtained in step 1.2 into the bidirectional long short-term memory network (Bi-LSTM) for spatiotemporal feature extraction and modeling, and obtain the auditory feature a with spatiotemporal information. lstm and visual features v lstm ; Step 1.4: Construct a multimodal similarity learning model and transform the auditory feature a with spatiotemporal information obtained in step 1.3 into lstm and visual features v lstm The input is into the multimodal similarity learning model, which uses the similarity graph between the visual segment features and the auditory segment features guided by the auditory features, the similarity graph of the intersection of the visual segment features and the auditory segment features guided by the auditory features, and the similarity graph of the combined features, and introduces feature random masks to generate similarity scores of segment features, model the semantic similarity of the audiovisual segments, select segments with high similarity scores, obtain segments with highly consistent semantics, and obtain visual features v with highly consistent semantics of the segments. MSLM and auditory features a MSLM ; Step 1.5: Build a global semantic perception and enhancement module; The global semantic perception and enhancement module consists of attention-based relation perception (Relation Aware), cross-relation perception (Audio-Video Interaction), and a bidirectional gated recurrent neural network (Bi-GRU). The global semantic perception and enhancement module perceives the video-level global semantics of a complete video in the temporal dimension through attention-based relation perception (Relation Aware) and cross-relation perception (Audio-Video Interaction); Attention-based relation awareness is expressed as: F out =Softmax(QK T )V, Q=FW Q ,K=FW K ,V=FW V , Based on cross-relationship perception (Audio-Video Interaction) it is expressed as: F va =F v ⊙F a F c =concat(F v ,F a ) q1=vW Q ,K 1,2 =m a,v IN K ,V 1,2 =F a,v IN V Among them, Q, K, V represent query, key, and value, and W Q ,W K ,W V represents the learnable parameter matrix, F represents the input features of attention-based relational perception, and F out represents the output features of attention-based relation perception, F v , F a represents the input features based on cross-relation perception, F av =AVI(v,a) represents the output features of cross-relation perception; The visual features v with highly consistent semantics of the fragments obtained in step 1.4 are MSLM and auditory features a MSLM Input into the global semantic perception and enhancement module, the visual features v with highly consistent semantics of the fragments MSLM and auditory features a MSLM First, it is transposed and input into the attention-based relational perception. The result is transposed again and input into the cross-relational perception, and the cross feature F is output. av ; Cross feature F av The video-level global semantics of the video is expressed, and the video-level global semantics of the video is introduced as a priori knowledge into the bidirectional gated recurrent neural network (Bi-GRU). The bidirectional gated recurrent neural network (Bi-GRU) transforms the cross feature F av As the common initial hidden state of two independent gated recurrent neural networks (GRU) to guide the dynamic modeling of visual and auditory features, and enhance the visual features v with highly consistent semantics of the segment obtained in step 1.4 MSLM and auditory features a MSLM The semantics of the enhanced fragment is obtained by highly consistent visual features V ISCE and auditory feature A ISCE ; This process means: THE ISCE =GRU(A seg ,F av ), In ISCE =GRU(V seg ,F av ), Step 1.6: The enhanced visual features V with highly consistent semantics of the fragments obtained in step 1.5 are ISCE and auditory feature A ISCE The fused features are input into the classification and recognition module to identify and locate the event categories in the target video.

3. The method for audiovisual event recognition and location based on semantically consistent segment selection according to claim 2, characterized in that: The implementation method of step 1.1 includes: Step 1.11: Get the target video The target video is decomposed into T continuous segments of 1 second in length. Each segment consists of a synchronized video signal segment and an audio signal segment. t V , S t A Represented as video signal segments and audio signal segments respectively; Step 1.12: Sample each video signal segment at a fixed number of frames per second to obtain a fixed number of images for each video signal segment. Use a convolutional neural network (VGG-19) to extract feature maps of the fixed number of images for each video signal segment. Globally merge the feature maps of the fixed number of images per second to obtain a global feature map, which is the segment-level feature of each video signal segment. Step 1.13: Convert each audio signal segment into a log-mel spectrogram and use a convolutional neural network (CNN) to extract segment-level features for each audio signal segment. Step 1.14: Arrange the segment-level features of each video signal segment extracted in step 1.12 and the segment-level features of each audio signal segment extracted in step 1.13 in temporal order to form the visual features of the video signal and the auditory features of the audio signal.

4. The method for audiovisual event recognition and location based on semantically consistent segment selection according to claim 2, characterized in that: The implementation method of step 1.4 includes: Step 1.41: Encode intramodal and cross-modal relationships between audiovisual modalities For the auditory feature a with spatiotemporal information obtained in step 1.3 lstm and visual features v lstm , firstly transform the auditory feature a lstm and visual features v lstm Connect to obtain the combined feature c lstm , and auditory feature a lstm , visual features v lstm and combined feature c lstm Performing linear transformation to obtain linearly transformed auditory segment features, visual segment features, and combined segment features; Step 1.42: Multimodal Similarity Learning: Transpose the auditory segment features after the linear transformation in step 1.41 and perform matrix multiplication with the visual segment features after the linear transformation to obtain the similarity map β between the visual segment features after the linear transformation and the auditory segment features. va ; The similarity graph β between the linearly transformed visual segment features and the auditory segment features va Perform the transposition operation to obtain the similarity graph β between the linearly transformed auditory segment features and visual segment features av , perform a transposition operation on the combined segment features after the linear transformation in step 1.41, and perform a matrix multiplication operation on the combined segment features after the linear transformation to obtain a similarity graph of the combined segment features; and perform a feature random masking operation on the similarity graph of the combined segment features, with a masking ratio of 0%-100%, to obtain a masked similarity graph of the combined segment features; The process of step 1.41 and step 1.42 is expressed by the following formula: Among them, β va , β av , β c , They represent the similarity graph between the visual segment features and the auditory segment features after linear transformation, the similarity graph between the auditory segment features and the visual segment features after linear transformation, the similarity graph of the combined segment features and the similarity graph of the combined segment features after masking, W1 v , W1 a , W1 c , represents the linear transformation operation, (·) T represents the transpose operation, d l Represents feature dimension; Step 1.43: Filtering of audiovisual modality consistent semantic segments: The thresholding method is used to obtain the similarity graph β between the linearly transformed visual segment features and the auditory segment features obtained in step 1.

42. va , the similarity graph β between the auditory segment features and the visual segment features after linear transformation av The visual segment features and auditory segment features with low similarity are connected and filtered, and only the visual segment features and auditory segment features with high semantic consistency are retained to obtain the similarity graph γ between the visual segment features and auditory segment features with high similarity. va Similarity graph between auditory segment features and visual segment features with high similarity γ av ; The above process is expressed as: c va =b va C(b va -t), c av =b av C(b av -t), Where τ represents a trainable hyperparameter and Γ(·) represents a thresholding operation that returns 1 if the input is greater than or equal to zero, otherwise it returns 0. Step 1.44: Perform matrix multiplication or dot product operations on different similarity graphs and different features to obtain the aggregation of audio-visual features; The specific method includes: lstm Similarity graph of combined fragment features after masking obtained in step 1.42 Perform a dot product operation and separate the results through the Split operation to obtain the auditory calibration feature a co and visual calibration features v co ; Compare the similarity graph γ between the linearly transformed visual segment features obtained in step 1.41 and the highly similar auditory segment features and visual segment features obtained in step 1.43 av Perform matrix multiplication to obtain visual features with high similarity; Compare the linearly transformed auditory segment features obtained in step 1.41 with the similarity graph γ between the visual segment features and auditory segment features with high similarity obtained in step 1.

43. va Perform matrix multiplication to obtain auditory features with high similarity; combine the visual features with high similarity with the visual calibration features v co Perform a dot product operation to combine the auditory features with high similarity with the auditory calibration features a co Perform a dot product operation and combine the two results with the auditory feature a with spatiotemporal information obtained in step 1.3 lstm and visual features v lstm Perform cross-addition to obtain visual features v with highly consistent semantics of the fragments MSLM and auditory features a MSLM ; The above process is expressed as: in co ,and co =Split(β c Fc lstm ), Among them, F represents the fully connected layer.

5. The method for audiovisual event recognition and location based on semantically consistent segment selection according to claim 2, characterized in that: The workflow of step 1.6 includes: Step 1.61: The enhanced visual features V with highly consistent semantics of the fragment obtained in step 1.5 ISCE and auditory feature A ISCE Obtain fusion features by adding features; Step 1.62: Input the fused features obtained in step 1.61 into the classification and recognition module. The classification and recognition module includes two fully connected layers. The audio-visual event recognition is performed based on the fused features to obtain the identification and location information of the target event.

6. The method for audiovisual event recognition and location based on semantically consistent segment selection according to claim 1, characterized in that: In step 2, the loss function is calculated as follows: according to different audiovisual event localization tasks, the loss function is divided into a loss function under full supervision and a loss function under weak supervision; The loss function in the fully supervised setting is composed of the event-related score s r , known segment-level event label Y tc , background label y t ={y t ||y t ∈{0,1},t=1,...,T} and the segment-level event labels O predicted by the model in the fully supervised setting tc Calculation, the calculation process is: OFF=W f (A ISCE +V ISCE ), s r =δ(AV·F1), s=A ISCE ⊙V ISCE / ||A ISCE ⊙V ISCE ||1 Where F1 represents the fully connected layer, AV represents the visual features with spatiotemporal information output by the Bi-LSTM (Bi-LSTM) and the audio features with spatiotemporal information are added together to obtain the fusion features, δ represents the activation function, and s represents the similarity vector of the segment-level visual and audio features; y tl represents y t The regularization result, L BCE (s r ,y t ) represents the event-related score s r and background label y t ={y t ||y t ∈{0,1},t=1,...,T} cross entropy loss, L MSE (s,y tl ) represents the similarity vectors s and y of segment-level visual and audio features t The regularization result y tl The mean square error of The loss function in the weakly supervised setting is the segment-level event labels predicted by the model in the weakly supervised setting. and video-level labels Y in a weakly supervised setting c Calculation, the calculation process is: Among them, s represents the softmax function and λ represents the hyperparameter.

7. An audiovisual event recognition and positioning system based on semantically consistent segment selection based on the method according to any one of claims 1 to 6, characterized in that: include: A model building module, used to build an audiovisual event recognition and localization model based on semantically consistent segment selection; The model training module is used to implement the designed loss function, and continuously train and optimize the audiovisual event recognition and localization model based on semantically consistent segment selection through the loss function. When the loss function is minimized, the optimal audiovisual event recognition and localization model based on semantically consistent segment selection is obtained; The recognition and positioning result output module is used to input the target video into the optimal audio-visual event recognition and positioning model based on semantically consistent segment selection to obtain the optimal target event recognition accuracy and target event positioning information.

8. An audiovisual event recognition and positioning device based on semantically consistent segment selection, characterized in that: include: Memory: a computer-readable device storing a computer program for the method for identifying and locating audiovisual events based on semantically consistent segment selection according to any one of claims 1 to 6; Processor: configured to implement the method for identifying and locating audiovisual events based on semantically consistent segment selection as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the audiovisual event recognition and positioning method based on semantically consistent segment selection as described in any one of claims 1 to 6.

Images