A method, device, medium and computer program product for locating audio and video events

By performing multiple spatial attention size compression and cross-modal learning on audio and video features, the interference of video features is eliminated and attention is enhanced in the relevant area, and the problem of failing to recognize fine-grained audio and video events in the prior art is solved, and the correct High-precision positioning of audio and video events.

CN118839308BActive Publication Date: 2024-12-03INSPUR (SHANDONG) COMPUTER TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411320978.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-23
Publication Date
2024-12-03
Estimated Expiration
2044-09-23

AI Technical Summary

Technical Problem

The existing audio and video event positioning technology can only achieve coarse-grained event recognition, and cannot identify fine-grained events such as different performance techniques in audio and video.

Method used

By compressing the initial video features multiple spatial attention sizes, a video compressed feature map of different sizes is obtained, and the probability distribution output function is used to calculate the attention weight of the video feature, and cross-modal learning is carried out in combination with audio features to achieve interest in the video area. Extraction of features and attention features, eliminating dimensions, performing spatial interference elimination processing of video features, enhancing spatial area attention related to candidate event types, and performing cross-modal learning to locate event types and start and end times.

Benefits of technology

It realizes a more fine-grained positioning of audio and video events, and can identify different performance techniques such as "continuous bow" and "jump bow" and other details, improving the accuracy and detailed recognition ability of event positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118839308B_ABST
    Figure CN118839308B_ABST
Patent Text Reader

Abstract

The present invention relates to artificial intelligence recognition technology, and specifically discloses an audio and video event positioning method, device, medium and computer program product, which calculates the corresponding video feature attention weight group by performing multiple spatial attention size compressions on the initial video features to obtain video compression feature maps of different sizes, and then reversely expands and superimposes the video feature attention weight group until it reaches the size of the initial video feature and then multiplies it with the initial video feature, and then performs spatial attention size compression on the interesting features of the video area to eliminate the size dimension, which not only realizes the conversion of video features into the same two-dimensional vector as audio features, but also enables the model to pay attention to more detailed information in space. The video attention features and audio features thus obtained are used for cross-modal learning to obtain the event type detection results and event start and end time detection results of the target audio and video files, thereby realizing more fine-grained audio and video event positioning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence recognition technology, and in particular to an audio and video event positioning method, device, medium and computer program product. Background Art

[0002] Audio-Visual Event Localization (AVEL) refers to the task of locating the temporal start and end positions of behaviors / events of a specified category in a given audio or video. It has a wide range of potential application scenarios in real life and has important application value.

[0003] However, current audio and video positioning solutions can only provide coarse-grained event recognition results. For example, they can identify the presence of "playing the violin" in the target audio and video, but cannot identify different playing techniques with details such as "legato bow" and "jumping bow".

[0004] How to achieve more fine-grained audio and video event positioning is a technical problem that technical personnel in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide an audio and video event positioning method, device, medium and computer program product for realizing more fine-grained audio and video event positioning.

[0006] In order to solve the above technical problems, the present invention provides an audio and video event positioning method, comprising:

[0007] The initial video features extracted from the target audio and video files are compressed multiple times using spatial attention to obtain video compression feature maps of different sizes;

[0008] The video feature attention weights at different spatial positions in the video compression feature map are calculated using a probability distribution output function to obtain a video feature attention weight group corresponding to the video compression feature map;

[0009] Starting from the video compression feature map with the smallest size, the corresponding video feature attention weight group is expanded to the size of the next video compression feature map, and then superimposed with the video feature attention weight group corresponding to the next video compression feature map, until it is expanded to the size of the initial video feature and then multiplied with the initial video feature to obtain the video region interest feature;

[0010] Performing multiple spatial attention size compressions on the feature of interest in the video region to eliminate the size dimension, thereby obtaining a video attention feature;

[0011] The video attention features and the audio features extracted from the target audio and video files are used for cross-modal learning to obtain event type detection results for the target audio and video files and event start and end time detection results for the target audio and video files.

[0012] On the one hand, the initial video features extracted from the target audio and video file are subjected to multiple spatial attention size compressions to obtain video compression feature maps of different sizes, including:

[0013] The initial video features are subjected to multiple spatial attention size compressions until the size value is compressed to 1, thereby obtaining the video compression feature maps of different sizes;

[0014] The feature of interest in the video region is subjected to multiple spatial attention size compressions to eliminate the size dimension, and the video attention feature is obtained, including:

[0015] The spatial attention size compression is performed multiple times on the feature of interest in the video region until the size value is compressed to 1, and then the size dimension is deleted to obtain the video attention feature.

[0016] On the other hand, cross-modal learning is performed using the video attention features and the audio features extracted from the target audio and video files, including:

[0017] Performing spatial interference elimination processing on the video feature according to the video attention feature to enhance the attention of the spatial region related to the candidate event type in the video frame to obtain a video interference elimination feature;

[0018] The video interference removal feature and the audio feature are used for cross-modal learning.

[0019] On the other hand, performing spatial interference elimination processing of the video feature according to the video attention feature to enhance the attention of the spatial region related to the candidate event type in the video frame to obtain the video interference elimination feature, including:

[0020] Reconstructing and encoding the spatial domain features of the video features according to the video attention features to obtain spatial domain coding features;

[0021] Calculate the confidence of the associated area of ​​the candidate event type based on the spatial domain coding feature to obtain a calculation result of the confidence of the associated area;

[0022] Perform classification prediction of the candidate event type based on the spatial domain coding features to obtain a confidence calculation result of the candidate event type;

[0023] The confidence calculation result of the associated region and the confidence calculation result of the candidate event type are used as the video interference elimination feature.

[0024] On the other hand, the confidence calculation result of the associated area of ​​the candidate event type is obtained based on the spatial domain coding feature, including:

[0025] The spatial coding features are input into the spatial prediction layer to transform and normalize the spatial coding features to obtain a size of The airspace prediction results of

[0026] The spatial domain prediction result is used as the confidence calculation result of the associated area;

[0027] in, Indicates the time, Indicates the image length, Indicates the image width, Indicates the number of candidate event types.

[0028] On the other hand, classification prediction of the candidate event type is performed based on the spatial domain coding feature to obtain a confidence calculation result of the candidate event type, including:

[0029] The spatial encoding feature is maximized in the middle dimension to obtain vector;

[0030] Will The vector is input into the non-spatial prediction layer to obtain the confidence calculation result of the candidate event type;

[0031] in, Indicates the time, Indicates the number of candidate event types.

[0032] On the other hand, performing spatial interference elimination processing of the video feature according to the video attention feature to enhance the attention of the spatial region related to the candidate event type in the video frame to obtain the video interference elimination feature, including:

[0033] Using a pre-trained spatial interference elimination module to perform spatial interference elimination processing on the video feature according to the video attention feature, so as to enhance attention on a spatial region related to the candidate event type in a video frame, and obtain the video interference elimination feature;

[0034] The pre-training step of the spatial interference elimination module includes:

[0035] Acquire a spatial interference elimination training data set; the spatial interference elimination training data set is a video data set including event type annotation information at the video frame level;

[0036] Based on the spatial interference elimination training data set, the spatial interference elimination module is subjected to loss optimization of event type classification results and associated region confidence calculation results to obtain the spatial interference elimination module.

[0037] On the other hand, the spatial interference elimination module is subjected to loss optimization of the event type classification result and the associated region confidence calculation result to obtain the spatial interference elimination module, including:

[0038] For the spatial prediction layer in the spatial interference elimination module, the loss function is used Conduct loss optimization processing;

[0039] For the non-spatial prediction layer in the spatial interference elimination module, a cross entropy loss function is used to perform loss optimization processing;

[0040] in, Represents the spatial prediction result at each moment, represents the maximum value in the spatial domain prediction result, Indicates that the label Convert to floating point number, Represents a logarithmic function.

[0041] On the other hand, cross-modal learning is performed using the video attention features and the audio features extracted from the target audio and video files to obtain event type detection results for the target audio and video files and event start and end time detection results for the target audio and video files, including:

[0042] Perform a cross-attention operation according to the video attention feature and the audio feature to obtain a video cross-attention feature and an audio cross-attention feature;

[0043] Splicing the video cross-attention feature and the audio cross-attention feature to obtain an audio-visual multimodal fusion encoding result;

[0044] The audio-visual multimodal fusion encoding result is input into a decoder, and classification calculation and regression calculation are performed respectively, and the event type detection result and the event start and end time detection result are output accordingly.

[0045] On the other hand, a cross-attention operation is performed according to the video attention feature and the audio feature to obtain a video cross-attention feature and an audio cross-attention feature, including:

[0046] Extracting features in the video features that are semantically aligned with the audio features according to the video attention features as video compression features, and features in the video features that are semantically non-aligned with the audio features as video special features, and splicing the video compression features and the video special features to obtain video splicing features;

[0047] Extracting features in the audio features that are semantically aligned with the video features as audio compression features, extracting features in the audio features that are semantically non-aligned with the video features as audio special features, and splicing the audio compression features and the audio special features to obtain audio splicing features;

[0048] Performing a cross-attention operation on the video splicing feature and the audio splicing feature to obtain the video cross-attention feature and the audio cross-attention feature;

[0049] The audio and video event positioning method also includes:

[0050] Extracting the video cross-attention special feature at a position corresponding to the video special feature in the video cross-attention feature and the audio cross-attention special feature at a position corresponding to the audio special feature in the audio cross-attention feature;

[0051] The audio-visual multimodal fusion encoding result is input into a decoder, and classification calculation and regression calculation are performed respectively, and the event type detection result and the event start and end time detection result are outputted correspondingly, including:

[0052] The audio-visual multimodal fusion encoding result, the video cross-attention special features and the audio cross-attention special features are input into the decoder, and classification calculation and regression calculation are performed respectively, and the event type detection result of the target audio and video file and the event start and end time detection result of the target audio and video file are correspondingly output.

[0053] On the other hand, a cross-attention operation is performed according to the video attention feature and the audio feature to obtain a video cross-attention feature and an audio cross-attention feature, which are calculated by the following formula:

[0054] ;

[0055] ;

[0056] in, Represents video features, Represents audio features, The cross-attention operation formula from audio modality to video modality is represented by: represents the query parameter matrix from the learnable audio modality to the video modality, represents the learnable key parameter matrix from the audio modality to the video modality, represents the learnable value parameter matrix from the audio modality to the video modality, Indicates the size of the audio feature, Indicates the duration of the complete video. The cross-attention operation formula from video modality to audio modality is represented by: represents the query parameter matrix from the learnable video modality to the audio modality, represents the key parameter matrix of the learnable video modality to audio modality, represents the learnable value parameter matrix from the video modality to the audio modality, Indicates the size of the video feature, Represents a normalization function, which is used to set the input number interval to between (-1,1).

[0057] On the other hand, the audio-visual multimodal fusion encoding result is input into the decoder, and classification calculation and regression calculation are performed respectively, and the event type detection result and the event start and end time detection result are outputted correspondingly, including:

[0058] Classification and regression calculations are performed using the following formula:

[0059] ;

[0060] ;

[0061] The event center of the event predicted at each moment is calculated as The start and end times of the event are and ;

[0062] The event center is used as the event type detection result, and the event start and end time is used as the event start and end time detection result;

[0063] in, Indicates the event type with the highest probability of occurring at each moment; Represents the learnable classification weight, which is , represents the number of video features, Indicates the number of candidate event types; represents the audio-visual multimodal fusion encoding result; Represents the index of the maximum value of the first dimension of the search vector; Represents the boundary regression parameter of the predicted time at each moment, and its size is , represents the moment, 2 represents two parameters that need to be regressed, one is It represents the ratio of the distance from the center of the event type identified at the current moment to the total video length. Indicates the ratio of the duration of the current event type to the unit duration of the current moment; represents the learnable regression weight; Represents a normalization function, which is used to set the input number interval to between (-1,1); Indicates the duration of the complete video; Indicates the frame number corresponding to the start and end time detection result of the event.

[0064] In order to solve the above technical problems, the present invention also provides an audio and video event positioning device, comprising:

[0065] Memory for storing computer programs;

[0066] The processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the audio and video event positioning method as described in any one of the above items are implemented.

[0067] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the audio and video event locating method as described in any one of the above items are implemented.

[0068] To solve the above technical problems, the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the audio and video event locating method as described in any one of the above items.

[0069] The audio and video event positioning method provided by the present invention has the beneficial effect that in the process of processing the video features extracted from the target audio and video file, the initial video features are compressed in spatial attention size multiple times to obtain video compression feature maps of different sizes; the video feature attention weights of different spatial positions in the video compression feature map are calculated using the probability distribution output function to obtain the video feature attention weight group corresponding to the video compression feature map; starting from the video compression feature map with the smallest size, the corresponding video feature attention weight group is expanded to the size of the next video compression feature map and then superimposed with the video feature attention weight group corresponding to the next video compression feature map until it is expanded to the size of the initial video feature and then multiplied with the initial video feature to obtain the video region interest feature; the video region interest feature is compressed in spatial attention size multiple times to eliminate the size dimension to obtain the video attention feature. This process not only realizes the conversion of video features into the same two-dimensional vector as the audio features extracted from the target audio and video file, but also realizes the attention enhancement at the video frame level, so that the model can pay attention to more detailed information in space. The obtained video attention features and audio features are used for cross-modal learning to obtain the event type detection results and event start and end time detection results of the target audio and video files, which can locate finer-grained events and achieve finer-grained audio and video event positioning.

[0070] The present invention also provides an audio and video event positioning device, a non-volatile storage medium and a computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 A flowchart of an audio and video event locating method provided by an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of the operation process of a multi-scale spatial attention module provided by an embodiment of the present invention;

[0074] Figure 3 A schematic diagram of a calculation process of a spatial interference elimination module provided by an embodiment of the present invention;

[0075] Figure 4 A schematic diagram of the operation process of an enhanced modal alignment module provided in an embodiment of the present invention;

[0076] Figure 5 A flowchart of another audio and video positioning method provided by an embodiment of the present invention;

[0077] Figure 6 A schematic diagram of the structure of an audio and video event locating device provided by an embodiment of the present invention;

[0078] Figure 7 A schematic diagram of the structure of an audio and video event locating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The core of the present invention is to provide an audio and video event positioning method, device, medium and computer program product for realizing more fine-grained audio and video event positioning.

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] Audio-Visual Event Localization (AVEL) refers to the task of locating the time start and end positions of the specified category of behavior / event in a given audio or video. The current implementation of audio-visual event localization tasks can usually only detect and locate coarse-grained events, but in many application scenarios, coarse-grained events cannot fully play a role, and it is necessary to identify each fine-grained event in the coarse-grained event. For example, in the application of learning violin instrument performance, not only the algorithm needs to recognize the event of "playing the violin", but also the performance techniques with different details such as "continuous bow" and "jumping bow". The performance technique is a more fine-grained event than "playing the violin"; in the application of diving competition scoring, not only the algorithm needs to recognize the event of "diving", but also each decomposed action needs to be identified to compare and score with the standard action. The decomposed action is a more fine-grained event than "diving".

[0082] In order to solve the problem of more fine-grained audio and video event positioning, an embodiment of the present invention provides an audio and video event positioning solution.

[0083] Figure 1 The present invention provides a flowchart of a method for locating an audio or video event.

[0084] like Figure 1 As shown, the audio and video event positioning method provided by the embodiment of the present invention includes: S101: performing multiple spatial attention size compressions on the initial video features extracted from the target audio and video file to obtain video compression feature maps of different sizes.

[0085] S102: Calculate the video feature attention weights at different spatial positions in the video compression feature map using a probability distribution output function to obtain a video feature attention weight group corresponding to the video compression feature map.

[0086] S103: Starting from the smallest video compression feature map, the corresponding video feature attention weight group is expanded to the size of the next video compression feature map, and then superimposed with the video feature attention weight group corresponding to the next video compression feature map, until it is expanded to the size of the initial video feature and then multiplied with the initial video feature to obtain the video area interest feature.

[0087] S104: Perform multiple spatial attention size compressions on the features of interest in the video region to eliminate the size dimension and obtain video attention features.

[0088] S105: Perform cross-modal learning using the video attention features and the audio features extracted from the target audio and video files to obtain event type detection results for the target audio and video files and event start and end time detection results for the target audio and video files.

[0089] There are many challenges in fine-grained audio and video event localization. First, the distinction of fine-grained events requires careful observation of details in the video rather than the overall picture, which is not available in current audio and video event localization solutions. For example, if you want to distinguish different techniques of "playing the violin", you need to focus on specific hand positions, which is a difficult point to consider in audio and video event localization solutions.

[0090] It can be seen that the distinction of fine-grained events usually requires observing the subtleties of the video. To this end, an embodiment of the present invention provides a multi-scale spatial attention module for performing video feature enhancement on the initial video features extracted from the target audio and video file to achieve the extraction of detail features.

[0091] In the embodiment of the present invention, it is necessary to extract synchronized video features and audio features from the target audio and video file, wherein the video features refer to features of the visual modality and the audio modality refers to features of the audio modality. The order of extraction is not specific.

[0092] Since the extracted audio features are two-dimensional vectors including time and audio feature size, in order to achieve cross-modal learning, the video features need to be converted into a two-dimensional vector form of time and video feature size.

[0093] In the embodiment of the present invention, the initial video features are converted from thought vectors to two-dimensional vectors by using spatial attention size compression.

[0094] In an embodiment of the present invention, in S101, the initial video features extracted from the target audio and video file are subjected to multiple spatial attention size compressions to obtain video compression feature maps of different sizes, which may include: the initial video features are subjected to multiple spatial attention size compressions until the size value is compressed to 1, to obtain video compression feature maps of different sizes.

[0095] In S104, multiple spatial attention size compressions are performed on the feature of interest in the video region to eliminate the size dimension and obtain the video attention feature. This may include: multiple spatial attention size compressions are performed on the feature of interest in the video region until the size value is compressed to 1, and then the size dimension is deleted to obtain the video attention feature.

[0096] Figure 2 A schematic diagram of the operation process of a multi-scale spatial attention module provided by an embodiment of the present invention. It should be noted that in the various drawings of the present invention, for easy distinction, the video features are represented by blue lines and the audio features are represented by orange lines.

[0097] like Figure 2 As shown, for S101, the size is The four-dimensional initial video features are sequentially input into the convolution layer, pooling layer, and nonlinear function layer (all three can use open source software) to obtain a size of After that, repeat this step until the two middle dimensions are compressed to 1, that is, the size is Video compression features. Remember that all the "intermediate" features are video compression feature maps, called video spatial feature groups (including The video compression characteristics of , excluding the size of Input features). Among them, Indicates the time, specifically corresponding to The moment of a video frame, , Indicates the video length corresponding to the video features extracted from the target audio and video file. It also indicates the total number of video frames extracted. Indicates the image length of the video frame, Indicates the image width of the video frame, Indicates the video feature size.

[0098] For ease of explanation, Figure 2 It is a diagram corresponding to the entire video and audio features, and its time mode is expressed as It represents the duration of the complete video; each feature block can represent the video features of one or more video frames, but it is only for illustration and does not limit the number of video frames.

[0099] Since in the embodiment of the present invention, multi-scale spatial attention processing is performed on the video frame, the first video frames and their corresponding ( The audio features of each video frame at the corresponding moment are introduced as the object.

[0100] like Figure 2 As shown, for S102, a probability distribution output function is used to perform spatial dimension operations on all features in the video spatial feature group. The probability distribution output function may include a normalized exponential function (softmax) and a normalization function to obtain the video feature attention weights at each position in space, and obtain the video feature attention weight group corresponding to the video compression feature map. Then, the smallest video feature attention weight is traversed, expanded (copied to twice the original size in the length and width spatial dimensions), and superimposed with the video feature attention weight of the next scale, and then expanded until the final vector is consistent with the initial feature size of the video, which is called the video area attention weight.

[0101] The initial video features are multiplied by the attention weights of the video regions to obtain the interesting features of the video regions. After that, for S104, the same parameters and structures as in S101 (several convolutional layers, pooling layers, and nonlinear function layers) can be used to obtain the final size of Features, cancel the two middle dimensions and get a size of Video attention features.

[0102] Spatial attention is usually a common technical means for optimizing or extracting visual features. However, since the final features of the visual backbone model have a low spatial dimension, the deep learning attention mechanism is usually used to process it, but this will make the network parameters larger, and its attention concentration effect is not ideal. The multi-scale spatial attention module designed in the embodiment of the present invention constructs a pseudo-process of pooling (from large to small in space) through cascaded softmax operations, normalization, and dimensional expansion operations. More global information can be used on the interface to assist the attention calculation of local information; and finally, the attention optimization of features is achieved through the reuse of isomorphic and isoparametric convolution models, that is, the convolution model provided by two open source software and the softmax function and normalization function that do not require training can be used. Then, the addition of a multi-scale spatial attention module almost does not increase the model training cost (or only needs to fine-tune the model parameters of the two sets of convolution models of the multi-scale spatial attention module) and the computational cost compared to the original audio and video positioning solution, and the optimization of the attention ability of the task to the video features is achieved, so that the model can pay attention to the spatial details information, and the gain effect of realizing a more fine-grained audio and video event positioning problem to be solved by the present invention is obvious. In essence, the effect of spatial attention is to enhance the attention to a certain area of ​​interest on the image and reduce the attention to other areas. When processing video features, it tends to enhance the attention to dynamic areas on the video frame (such as people and even people's hands) and reduce the attention to static areas (such as background). In the audio and video positioning method provided in the embodiment of the present invention, after S101~S104, the corresponding Figure 2 The three-column operation process in is equivalent to performing three video frame-level spatial region of interest optimizations, and finally we can get Figure 2 As shown in the upper right corner of the figure, the video attention feature significantly focuses on the area of ​​the character's hands and weakens other areas. This helps to pay more attention to the area in each video frame that is conducive to fine-grained audio and video event positioning in the subsequent cross-modal learning of audio and video features.

[0103] It should be noted that S101 and S102 may not be in a completely sequential calculation relationship, that is, S101 may be executed first and then S102; or, in the process of executing S101, each time a video compression feature map is obtained, the calculation of the video feature attention weight group of the video compression feature map in S102 is executed accordingly, and after obtaining the video compression feature map with the smallest size, all the required video feature attention weight groups are also obtained.

[0104] For S105, the video attention features obtained by processing the multi-scale spatial attention module and the audio features extracted from the target audio and video files are used for cross-modal learning. By using the video attention features that pay more attention to the detailed regional features in each video frame, a more fine-grained audio and video event positioning result can be obtained, which is shown in the form of event type detection results for the target audio and video files and event start and end time detection results for the target audio and video files. Among them, the event type detection result can be an event type selected from the input candidate event types.

[0105] The audio and video event positioning method provided by the embodiment of the present invention performs multiple spatial attention size compression on the initial video features in the process of processing the video features extracted from the target audio and video file to obtain video compression feature maps of different sizes; the video feature attention weights of different spatial positions in the video compression feature map are calculated using the probability distribution output function to obtain the video feature attention weight group corresponding to the video compression feature map; starting from the video compression feature map with the smallest size, the corresponding video feature attention weight group is expanded to the size of the next video compression feature map and then superimposed with the video feature attention weight group corresponding to the next video compression feature map until it is expanded to the size of the initial video feature and then multiplied with the initial video feature to obtain the video region interest feature; the video region interest feature is compressed multiple times in spatial attention size to eliminate the size dimension to obtain the video attention feature. This process not only realizes the conversion of video features into the same two-dimensional vector as the audio features extracted from the target audio and video file, but also realizes the attention enhancement at the video frame level, so that the model can pay attention to more detailed information in space. The obtained video attention features and audio features are used for cross-modal learning to obtain the event type detection results and event start and end time detection results of the target audio and video files, which can locate finer-grained events and achieve finer-grained audio and video event positioning.

[0106] Based on the above embodiments, in the audio and video positioning method provided in the embodiments of the present invention, cross-modal learning is performed using video attention features and audio features extracted from the target audio and video files in S105, which may include: performing spatial interference elimination processing on video features according to the video attention features to enhance attention on the spatial area related to the candidate event type in the video frame to obtain video interference elimination features; and using video interference elimination features and audio features for cross-modal learning.

[0107] Since fine-grained events are usually tiny and easily interfered by other events, the embodiment of the present invention designs a spatial interference elimination module to further eliminate the interference of video features irrelevant to the audio and video events to be located.

[0108] Figure 3 A schematic diagram of the operation process of a spatial interference elimination module provided in an embodiment of the present invention.

[0109] like Figure 3 As shown, in some optional implementations of the embodiments of the present invention, spatial interference elimination processing of video features is performed according to video attention features to enhance attention on spatial regions related to candidate event types in video frames to obtain video interference elimination features, which may include: spatial domain feature reconstruction and encoding processing of video features according to video attention features to obtain spatial domain coding features; associated region confidence calculation of candidate event types based on spatial domain coding features to obtain associated region confidence calculation results; classification prediction of candidate event types based on spatial domain coding features to obtain confidence calculation results of candidate event types; and using the associated region confidence calculation results and the candidate event type confidence calculation results as video interference elimination features.

[0110] For ease of explanation, Figure 3 It is a diagram corresponding to the entire video and audio features, and its time mode is expressed as It represents the duration of the complete video; each feature block can represent the video features of one or more video frames, but it is only for illustration and does not limit the number of video frames.

[0111] Since in the embodiment of the present invention, the spatial interference elimination process is performed on the video frame, the first video frames and their corresponding ( The audio features of each video frame at the corresponding moment are introduced as the object.

[0112] In cross-modal learning, video features are converted into two dimensions due to alignment with two-dimensional audio features, resulting in loss of some spatial features or neglect of details. Therefore, the purpose of designing a spatial interference elimination module in an embodiment of the present invention is to allow events to supervise spatial features on video frames. For example, the event "left-hand technique" focuses on supervising the left-hand area.

[0113] In an embodiment of the present invention, the spatial domain feature reconstruction and encoding processing of the video feature are performed according to the video attention feature to obtain the spatial domain encoding feature, which may include: The video enhancement features of are input into the spatial dimension reconstruction module and reconstructed into a size of The spatial domain features of . Among them, the size can be The video features are input into the perception layer, and the perception layer formula is: ,in Indicates size The learnable perceptual weight matrix, Indicates the preset spatial dimension size, Indicates size The learnable perceptual bias vector. The reconstruction layer formula is ,in Indicates size The learnable reconfigurable weight matrix. Size Afterwards, the output features are reconstructed into The form of is called spatial domain feature.

[0114] The spatial features are encoded using the perception layer (multi-layer perceptron MLP can be used) to obtain a size of The spatial encoding features of .

[0115] Calculating the confidence of the associated region of the candidate event type based on the spatial coding feature to obtain the confidence calculation result of the associated region may include: inputting the spatial coding feature into the spatial prediction layer to transform and normalize the spatial coding feature to obtain a size of The airspace prediction result of ; the airspace prediction result is used as the confidence calculation result of the associated area; among them, Indicates the time, Indicates the image length, Indicates the image width, Indicates the number of candidate event types. A perception layer can be used to transform the spatial encoding features, and an activation function (sigmoid) can be used to normalize the output nonlinearly. After that, the spatial prediction results are input into the decoder for inference.

[0116] Based on the spatial domain coding features, the classification prediction of the candidate event type is performed to obtain the confidence calculation result of the candidate event type, which may include: taking the maximum value of the spatial domain coding feature in the middle dimension to obtain vector; The vector is input into the non-spatial prediction layer to obtain the confidence calculation result of the candidate event type; among them, Indicates the time, Indicates the number of candidate event types. Afterwards, the non-spatial domain prediction results are input into the decoder for inference.

[0117] In the embodiment of the present invention, the role of the spatial interference elimination module is to find the areas associated with the candidate event types in the video frame and determine the degree of association between them and the candidate event types, which respectively correspond to the non-spatial prediction layer and the spatial prediction layer introduced above, that is, the essence of the spatial prediction layer is to give the attention area corresponding to the candidate event type (such as the left hand, right hand, left foot, and right foot of the character). If there are multiple attention areas, the probability that they may be associated with the candidate event type is given, that is, the confidence calculation result of the associated area. The non-spatial prediction layer is used to determine whether there is a candidate event type on the current video frame. If there are c types of candidate event types, the probability corresponding to each type of candidate event type is given, which is the confidence calculation result of the candidate event type.

[0118] Assume that the candidate event types have Class, recognizes the fine-grained event of "tremolo action", corresponding to the If there are five associated regions, the spatial prediction layer can only determine the maximum value (whether it is 1) in the confidence calculation result of the associated region, and the other associated regions or even other regions of the video frame can be ignored. In the non-spatial prediction layer, the judgment result of whether the candidate event type exists is output, that is, the confidence calculation result of the candidate event type.

[0119] That is to say, the non-spatial prediction layer gives the judgment result of whether the candidate event type exists, and the spatial prediction layer gives a more fine-grained judgment basis. The final output result is essentially the non-spatial prediction result, that is, it indicates whether a fine-grained event does exist in the current video frame.

[0120] If the model determines that there are multiple possible event types in the current video frame, it will output multiple sets of confidence calculation results for associated regions and confidence calculation results for candidate event types.

[0121] In the embodiment of the present invention, there is no order in which the spatial prediction layer is calculated and the non-spatial prediction layer is calculated.

[0122] In an embodiment of the present invention, performing spatial interference elimination processing on video features according to video attention features to enhance attention on spatial regions in video frames that are related to candidate event types to obtain video interference elimination features may include: using a pre-trained spatial interference elimination module to perform spatial interference elimination processing on video features according to video attention features to enhance attention on spatial regions in video frames that are related to candidate event types to obtain video interference elimination features.

[0123] The pre-training step of the spatial interference elimination module may include: obtaining a spatial interference elimination training data set; the spatial interference elimination training data set is a video data set including event type annotation information at the video frame level; based on the spatial interference elimination training data set, the spatial interference elimination module is subjected to loss optimization of event type classification results and associated area confidence calculation results to obtain the spatial interference elimination module.

[0124] When training the spatial interference elimination module, the method of annotating video frames can be used to achieve multi-event learning, that is, only the events corresponding to the video frame are annotated, and the spatial interference elimination module is trained to focus on the areas in the video frame that are associated with the candidate event types and determine their degree of association with the candidate event types.

[0125] Then, the spatial interference elimination module is subjected to loss optimization of the event type classification result and the associated area confidence calculation result to obtain the spatial interference elimination module, which may include: for the spatial domain prediction layer in the spatial interference elimination module, using the loss function Perform loss optimization processing; for the non-spatial prediction layer in the spatial interference elimination module, use the cross-entropy loss function (Cross-Entropy) to perform loss optimization processing. Among them, Represents the spatial prediction result at each moment, Represents the maximum value in the spatial domain prediction results, Indicates that the label Convert to floating point number to facilitate multiplication with the predicted floating point number; represents the logarithmic function, which is the likelihood function used in this formula.

[0126] On the basis of the above embodiments, in the audio and video positioning method provided in the embodiments of the present invention, S105 uses video attention features and audio features extracted from the target audio and video files to perform cross-modal learning to obtain event type detection results for the target audio and video files and event start and end time detection results for the target audio and video files, which may include: performing cross-attention operations based on video attention features and audio features to obtain video cross-attention features and audio cross-attention features; splicing video cross-attention features and audio cross-attention features to obtain audio-visual multimodal fusion coding results; inputting the audio-visual multimodal fusion coding results into a decoder, performing classification calculations and regression calculations respectively, and outputting event type detection results and event start and end time detection results accordingly.

[0127] In the target audio and video files, audio features and video features are complementary to a certain extent, that is, actions and sounds may occur at the same time, but some events may be covered by noise in the audio but can be clearly observed in the video, and some events cannot be seen in the video but can be heard through the audio. Therefore, it is necessary to design a reasonable audio and video complementary solution.

[0128] Figure 4 A schematic diagram of the operation process of an enhanced modal alignment module provided in an embodiment of the present invention.

[0129] In an embodiment of the present invention, cross-attention operations are performed according to video attention features and audio features to obtain video cross-attention features and audio cross-attention features, which may include: extracting features in video features that are semantically aligned with audio features as video compression features, and features in video features that are semantically non-aligned with audio features as video special features according to the video attention features, and splicing the video compression features and the video special features to obtain video splicing features; extracting features in audio features that are semantically aligned with video features as audio compression features, and features in audio features that are semantically non-aligned with video features as audio special features, and splicing the audio compression features and the audio special features to obtain audio splicing features; performing cross-attention operations on video splicing features and audio splicing features to obtain video cross-attention features and audio cross-attention features.

[0130] On this basis, the audio and video positioning method provided by the embodiment of the present invention may also include: extracting the video cross-attention special features at the position corresponding to the video special features in the video cross-attention features and the audio cross-attention special features at the position corresponding to the audio special features in the audio cross-attention features. Inputting the audio-visual multimodal fusion encoding result into the decoder, performing classification calculation and regression calculation respectively, and correspondingly outputting the event type detection result and the event start and end time detection result, may include: inputting the audio-visual multimodal fusion encoding result, the video cross-attention special features and the audio cross-attention special features into the decoder, performing classification calculation and regression calculation respectively, and correspondingly outputting the event type detection result of the target audio and video file and the event start and end time detection result of the target audio and video file.

[0131] For example, when identifying the fine-grained event "tremolo action" in the coarse-grained event "playing violin", there will be a situation where the audio modality and the video modality are not semantically aligned. At this time, the occurrence of the "tremolo action" event can be more accurately judged by the audio modality, and the start and end time of the "tremolo action" event can be more accurately judged by the video modality. In the recognition process of other events, it may be just the opposite, that is, the start and end time of the event can be accurately located by the audio modality, and the event type can be accurately located by the video modality. That is to say, sometimes one of the audio modality and the video modality may carry more features that are more useful for event classification, while the other modality carries more features that are more useful for event start and end time positioning. Therefore, in an embodiment of the present invention, while the audio modality and the video modality are semantically aligned across modalities (obtaining video compression features and audio compression features), those unaligned parts (i.e., video special features and audio special features) are also retained, and based on the video special features and the audio special features to assist the event type classification task and the event start and end time positioning task, it is determined whether the two special features are more conducive to the classification task or the regression task, so as to achieve enhanced audio and video modality alignment.

[0132] To implement the above steps, an embodiment of the present invention designs an enhanced modality alignment module for performing special feature extraction, special feature and compressed feature concatenation, and cross-attention operations.

[0133] For ease of explanation, Figure 4 It is a diagram corresponding to the entire video and audio features, and its time mode is expressed as It represents the duration of the complete video; each feature block can represent the video features of one or more video frames, but it is only for illustration and does not limit the number of video frames.

[0134] Since in the embodiment of the present invention, the video frame is processed, the first video frames and their corresponding ( The audio features of each video frame at the corresponding moment are introduced as the object.

[0135] like Figure 4 As shown, the size is The video attention features are input into the video modality special representation layer and feature compression layer, and are converted into a size of The video has special features and size The audio features are input into the audio modality special representation layer and feature compression layer, and the learnable parameters are used to convert them into a size of The audio characteristics and size of Audio compression features.

[0136] The video compression features are concatenated with the video special features to obtain a size of The video splicing features of . The audio compression features are spliced ​​with the audio special features to obtain a size of Audio splicing features.

[0137] Cross-attention operations are performed on the video splicing features and the audio splicing features to obtain video cross-attention features and audio cross-attention features whose sizes are consistent with the input.

[0138] According to the video attention features and audio features, cross-attention operations are performed to obtain the video cross-attention features and audio cross-attention features, which are calculated by the following formula:

[0139] ;

[0140] ;

[0141] Among them, the first formula is the cross-attention formula from the audio modality to the video modality, and the second formula is the cross-attention formula from the video modality to the audio modality. Represents video features, Represents audio features, The cross-attention operation formula from audio modality to video modality is represented by: represents the query parameter matrix from the learnable audio modality to the video modality, represents the learnable key parameter matrix from the audio modality to the video modality, represents the learnable value parameter matrix from the audio modality to the video modality, Indicates the size of the audio feature, Indicates the duration of the complete video. The cross-attention operation formula from video modality to audio modality is represented by: represents the query parameter matrix from the learnable video modality to the audio modality, represents the key parameter matrix of the learnable video modality to audio modality, represents the learnable value parameter matrix from the video modality to the audio modality, Indicates the size of the video feature, Represents a normalization function, which is used to set the input number interval to between (-1,1).

[0142] In practical applications, this cross-attention operation process is repeated times, such as Figure 4 shown.

[0143] The features of the corresponding positions of the special modal special representations in the video cross-attention features and the audio cross-attention features are separated out, namely the video cross-attention special features and the audio cross-attention special features, and respectively input into the decoders mentioned above (including regression and recognition functions) to assist the mainstream features (i.e. the audio-visual multimodal fusion encoding results) for decoding.

[0144] The learnable parameter matrix in the enhanced modal alignment module provided in the embodiment of the present invention needs to be obtained by pre-training using an audio and video dataset.

[0145] Figure 5 A flowchart of another audio and video positioning method provided by an embodiment of the present invention.

[0146] On the basis of the above embodiments, in an embodiment of the present invention, cross-attention operations are performed according to video attention features and audio features to obtain video cross-attention features and audio cross-attention features, which may include: performing feature embedding processing on the video attention features and the initial audio features respectively to encode the video attention features and the initial audio features into a unified dimension to obtain video embedding features and audio embedding features; performing cross-attention operations on the video embedding features and the audio embedding features to obtain audio-visual multimodal fusion encoding results.

[0147] On this basis, Figure 5 The process of another audio and video positioning method provided by an embodiment of the present invention is shown, and the multi-scale spatial attention module, enhanced modal alignment module and spatial interference elimination module introduced in the above embodiment are used to obtain a more fine-grained audio and video event positioning result.

[0148] For ease of explanation, Figure 5 It is a diagram corresponding to the entire video and audio features, and its time mode is expressed as It represents the duration of the complete video; each feature block can represent the video features of one or more video frames, but it is only for illustration and does not limit the number of video frames.

[0149] Since in the embodiment of the present invention, the video frame is processed, the first video frames and their corresponding ( The audio features of each video frame at the corresponding moment are introduced as the object.

[0150] like Figure 5 As shown, the video content in the target audio and video file can be extracted and input into the video feature extractor to obtain the initial video feature. The audio content in the target audio and video file can be extracted and input into the audio feature extractor to obtain the initial audio feature.

[0151] The video feature extractor and the audio feature extractor can use existing models, such as the deep learning model I3D (Inception Inflated 3D ConvNet) for video feature extraction, the self-supervised learning VideoMAEv2 model for video feature extraction, etc. The audio feature extractor can use the VGGish model, etc.

[0152] As described in the above embodiment of the present invention, from the perspective of feature dimension, the initial audio feature is a two-dimensional vector ( The first dimension represents the time, and the second dimension represents the audio feature size), but the initial video feature is a four-dimensional vector ( The first dimension represents the time, the second and third dimensions represent the length and width of the image, and the fourth dimension represents the size of the video feature). Therefore, the initial video feature is input into the multi-scale spatial attention module introduced in the above embodiment of the present invention, and the video attention feature is output after encoding. The video attention feature is a feature that selects more valuable detail areas from the initial video feature and reduces the dimension to two dimensions. The size is .

[0153] The video attention features and the initial audio features are input into the feature embedding layer respectively to obtain the video embedding features and the audio embedding features. The calculation formula of the feature embedding layer can be: .in, , , They are the learnable query parameter matrix, the learnable key parameter matrix, and the learnable value parameter matrix. is the feature input to the feature embedding layer, represents the dimension of the input features, is the learnable embedding matrix, is a learnable bias vector. The feature embedding layer is used to encode different vectors into a unified dimension for subsequent processing. The two feature sizes output by the feature embedding layer are .

[0154] The video embedding features and audio embedding features are input into the enhanced modality alignment module to obtain video enhancement features and audio enhancement features. The sizes of both features are . Then, the enhanced modal alignment module provided in the above embodiment of the present invention is used to perform cross-attention operations according to the video attention features and the audio features to obtain video cross-attention features and audio cross-attention features, which may include: extracting features in the video embedded features that are semantically aligned with the audio embedded features as video compression features, and extracting features in the video embedded features that are semantically non-aligned with the audio embedded features as video special features, splicing the video compression features and the video special features to obtain video splicing features; extracting features in the audio embedded features that are semantically aligned with the video embedded features as audio compression features, and extracting features in the audio embedded features that are semantically non-aligned with the video embedded features as audio special features, splicing the audio compression features and the audio special features to obtain audio splicing features; performing cross-attention operations on the video splicing features and the audio splicing features to obtain video cross-attention features and audio cross-attention features. Please refer to the introduction of the above embodiment for the remaining steps.

[0155] The video enhancement feature is input into the spatial interference elimination module to obtain the video interference elimination feature, which is of size . The spatial interference elimination module provided in the above embodiment of the present invention is used to perform spatial interference elimination processing on video features according to video attention features to enhance the attention of spatial regions related to candidate event types in video frames to obtain video interference elimination features, which may include: reconstructing and encoding video enhancement features to obtain spatial domain coding features; calculating the confidence of associated regions of candidate event types based on spatial domain coding features to obtain calculation results of associated region confidence; performing classification prediction on candidate event types based on spatial domain coding features to obtain calculation results of confidence of candidate event types; and using the calculation results of associated region confidence and the confidence calculation results of candidate event types as video interference elimination features. For the remaining steps, please refer to the introduction of the above embodiment.

[0156] Combine video noise removal features and audio enhancement features into one The multimodal fusion features, namely the audio-visual multimodal fusion encoding result mentioned in the above embodiment, are then input into the decoder for two groups of predictions. The decoder includes two classification calculation functions and a regression calculation function.

[0157] The audio-visual multimodal fusion encoding result is input into the decoder, and classification calculation and regression calculation are performed respectively, and the event type detection result and the event start and end time detection result are output accordingly, including: classification calculation and regression calculation are performed by the following formula:

[0158] ;

[0159] ;

[0160] The event center of the event predicted at each moment is calculated as The start and end times of the event are and ;

[0161] The event center is used as the event type detection result, and the event start and end time is used as the event start and end time detection result.

[0162] in, Indicates the event type with the highest probability of occurring at each moment; Represents the learnable classification weight, which is , represents the number of video features, Indicates the number of candidate event types; Represents the audio-visual multimodal fusion encoding result; Represents the index of the maximum value of the first dimension of the search vector; Represents the boundary regression parameter of the predicted time at each moment, and its size is , represents the moment, 2 represents two parameters that need to be regressed, one is It represents the ratio of the distance from the center of the event type identified at the current moment to the total video length. Indicates the ratio of the duration of the current event type to the unit duration of the current moment; represents the learnable regression weight; Represents a normalization function, which is used to set the input number interval to between (-1,1); Indicates the duration of the complete video; Indicates the frame number corresponding to the event start and end time detection result.

[0163] It should be noted that in the embodiments of the audio and video event locating method of the present invention, some of the steps or features can be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the audio and video event locating method provided in the embodiments of the present invention.

[0164] The above describes in detail various embodiments corresponding to the audio and video event locating method. On this basis, the present invention also discloses an audio and video event locating device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0165] Figure 6 A schematic diagram of the structure of an audio and video event locating device provided by an embodiment of the present invention.

[0166] like Figure 6 As shown, the audio and video event locating device provided by the embodiment of the present invention includes:

[0167] A first spatial compression unit 601 is used to perform multiple spatial attention size compressions on the initial video features extracted from the target audio and video file to obtain video compression feature maps of different sizes;

[0168] The attention weight calculation unit 602 is used to calculate the video feature attention weights at different spatial positions in the video compression feature map by using a probability distribution output function to obtain a video feature attention weight group corresponding to the video compression feature map; starting from the video compression feature map with the smallest size, the corresponding video feature attention weight group is expanded to the size of the next video compression feature map, and then superimposed with the video feature attention weight group corresponding to the next video compression feature map, until it is expanded to the size of the initial video feature and then multiplied with the initial video feature to obtain a feature of interest in the video region;

[0169] A second spatial compression unit 603 is used to perform multiple spatial attention size compressions on the feature of interest in the video region to eliminate the size dimension and obtain a video attention feature;

[0170] The cross-modal computing unit 604 is used to perform cross-modal learning using video attention features and audio features extracted from the target audio and video files to obtain event type detection results for the target audio and video files and event start and end time detection results for the target audio and video files.

[0171] Among them, the first spatial compression unit 601, the attention weight calculation unit 602 and the second spatial compression unit 603 can refer to the multi-scale spatial attention module introduced in the above embodiment of the present invention.

[0172] The cross-modal computing unit 604 may include:

[0173] A feature embedding unit, comprising a video feature embedding layer and an audio feature embedding layer, for respectively performing feature embedding processing on the video attention feature and the initial audio feature, so as to encode the video attention feature and the initial audio feature into a unified dimension, thereby obtaining a video embedding feature and an audio embedding feature;

[0174] An enhanced modality alignment module is used to perform cross-modal enhanced learning based on video embedding features and audio embedding features, and output video enhanced features and audio enhanced features;

[0175] A spatial interference elimination module is used to perform spatial interference elimination processing on the video enhancement feature to enhance the attention of the spatial region related to the candidate event type in the video frame to obtain the video interference elimination feature;

[0176] A feature splicing unit, used for splicing video interference elimination features and audio enhancement features to obtain an audio-visual multimodal fusion coding result;

[0177] The decoder is used to decode the audio-visual multimodal fusion encoding results, including classification calculation and regression calculation, and correspondingly obtain the event type detection results of the target audio and video files and the event start and end time detection results of the target audio and video files.

[0178] It should be noted that in each implementation of the audio and video event locating device provided in the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be adopted. The connection method between different units can be electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0179] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0180] Figure 7 A schematic diagram of the structure of an audio and video event locating device provided by an embodiment of the present invention.

[0181] like Figure 7 As shown, the audio and video event locating device provided by the embodiment of the present invention includes: a memory 710, used to store a computer program 711; a processor 720, used to execute the computer program 711, and when the computer program 711 is executed by the processor 720, the steps of the audio and video event locating method provided in any of the above embodiments are implemented.

[0182] Among them, the processor 720 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 720 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 720 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 720 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 720 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0183] The memory 710 may include one or more non-volatile storage media, which may be non-transitory. The memory 710 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 710 is at least used to store the following computer program 711, wherein the computer program 711, after being loaded and executed by the processor 720, can implement the relevant steps in the audio and video event positioning method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 710 may also include an operating system 712 and data 713, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 712 may be Windows or other types of operating systems. Data 713 may include but is not limited to the data involved in the above method.

[0184] In some embodiments, the audio and video event locating device may further include a display screen 730 , a power supply 740 , a communication interface 750 , an input and output interface 760 , a sensor 770 , and a communication bus 780 .

[0185] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the audio and video event locating device, and may include more or less components than those shown in the figure.

[0186] The audio and video event locating device provided in the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the audio and video event locating method provided in the above embodiment, and the effect is the same as above.

[0187] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the audio and video event locating method provided in any one of the above embodiments can be implemented.

[0188] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0189] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the audio and video event positioning method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0190] An embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the audio and video event locating method provided in any one of the above embodiments.

[0191] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the audio and video event positioning method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0192] The above is a detailed introduction to an audio and video event positioning method, device, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0193] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for locating audio and video events, characterized in that: include: The initial video features extracted from the target audio and video files are compressed multiple times by spatial attention until the size value is compressed to 1, and video compression feature maps of different sizes are obtained; The video feature attention weights at different spatial positions in the video compression feature map are calculated using a probability distribution output function to obtain a video feature attention weight group corresponding to the video compression feature map; Starting from the video compression feature map with the smallest size, the corresponding video feature attention weight group is expanded to the size of the next video compression feature map, and then superimposed with the video feature attention weight group corresponding to the next video compression feature map, until it is expanded to the size of the initial video feature and then multiplied with the initial video feature to obtain the video region interest feature; Performing multiple spatial attention size compressions on the feature of interest in the video region until the size value is compressed to 1, deleting the size dimension, and obtaining a video attention feature; Using the video attention features and the audio features extracted from the target audio and video files to perform cross-modal learning, obtain event type detection results for the target audio and video files and event start and end time detection results for the target audio and video files; Among them, cross-modal learning is performed using the video attention features and the audio features extracted from the target audio and video files to obtain event type detection results for the target audio and video files and event start and end time detection results for the target audio and video files, including: Perform a cross-attention operation according to the video attention feature and the audio feature to obtain a video cross-attention feature and an audio cross-attention feature; Performing spatial interference elimination processing on video features according to the video cross-attention features to enhance attention on spatial regions related to the candidate event types in the video frame to obtain video interference elimination features; Splicing the video interference elimination feature and the audio cross-attention feature to obtain an audio-visual multimodal fusion encoding result; Input the audio-visual multimodal fusion encoding result into a decoder, perform classification calculation and regression calculation respectively, and output the event type detection result and the event start and end time detection result accordingly; The spatial interference elimination processing of the video feature is performed according to the video cross-attention feature to enhance the attention of the spatial region related to the candidate event type in the video frame to obtain the video interference elimination feature, including: Perform spatial domain feature reconstruction and encoding processing of video features according to the video cross-attention features to obtain spatial domain coding features; The spatial coding features are input into the spatial prediction layer to transform and normalize the spatial coding features to obtain a size of The prediction result of the airspace is used as the confidence calculation result of the associated area; The spatial encoding feature is maximized in the middle dimension to obtain vector; The vector is input into the non-spatial domain prediction layer to perform classification prediction of the candidate event type, and obtain a confidence calculation result of the candidate event type; The confidence calculation result of the associated region and the confidence calculation result of the candidate event type are used as the video interference elimination feature; The audio-visual multimodal fusion encoding result is input into the decoder, and classification calculation and regression calculation are performed respectively, and the event type detection result and the event start and end time detection result are outputted correspondingly, including: Input the audio-visual multimodal fusion encoding result into a decoder, perform classification calculation and regression calculation respectively, and output the boundary regression parameters of the event type with the highest probability of occurrence at each moment and the predicted time at each moment; The event center of the event predicted at each moment is calculated as The start and end times of the event are and ; The event center is used as the event type detection result, and the event start and end time is used as the event start and end time detection result; in, Indicates the time, Indicates the image length, Indicates the image width, represents the number of candidate event types, Indicates the duration of the complete video; Indicates the number of frames corresponding to the detection result of the start and end time of the event, Indicates the ratio of the distance from the moment center of the event type identified at the current moment to the total video length. Indicates the ratio of the duration of the current event type to the unit duration of the current moment.

2. The method for locating audio and video events according to claim 1, characterized in that: The spatial interference elimination processing of the video feature is performed according to the video cross-attention feature to enhance the attention of the spatial region related to the candidate event type in the video frame to obtain the video interference elimination feature, including: Using a pre-trained spatial interference elimination module to perform spatial interference elimination processing on the video feature according to the video cross-attention feature, so as to enhance the attention of the spatial region related to the candidate event type in the video frame, and obtain the video interference elimination feature; The pre-training step of the spatial interference elimination module includes: Acquire a spatial interference elimination training data set; the spatial interference elimination training data set is a video data set including event type annotation information at the video frame level; Based on the spatial interference elimination training data set, the spatial interference elimination module is subjected to loss optimization of event type classification results and associated region confidence calculation results to obtain the spatial interference elimination module.

3. The method for locating audio and video events according to claim 2, characterized in that: The spatial interference elimination module is subjected to loss optimization of the event type classification result and the associated region confidence calculation result to obtain the spatial interference elimination module, including: For the spatial prediction layer in the spatial interference elimination module, the loss function is used Conduct loss optimization processing; For the non-spatial prediction layer in the spatial interference elimination module, a cross entropy loss function is used to perform loss optimization processing; in, Represents the spatial prediction result at each moment, represents the maximum value in the spatial domain prediction result, Indicates that the label Convert to floating point number, Represents a logarithmic function.

4. The method for locating audio and video events according to claim 1, characterized in that: A cross-attention operation is performed according to the video attention feature and the audio feature to obtain a video cross-attention feature and an audio cross-attention feature, including: Extracting features in the video features that are semantically aligned with the audio features according to the video attention features as video compression features, and features in the video features that are semantically non-aligned with the audio features as video special features, and splicing the video compression features and the video special features to obtain video splicing features; Extracting features in the audio features that are semantically aligned with the video features as audio compression features, extracting features in the audio features that are semantically non-aligned with the video features as audio special features, and splicing the audio compression features and the audio special features to obtain audio splicing features; Performing a cross-attention operation on the video splicing feature and the audio splicing feature to obtain the video cross-attention feature and the audio cross-attention feature; The audio and video event positioning method also includes: Extracting the video cross-attention special feature at a position corresponding to the video special feature in the video cross-attention feature and the audio cross-attention special feature at a position corresponding to the audio special feature in the audio cross-attention feature; The audio-visual multimodal fusion encoding result is input into a decoder, and classification calculation and regression calculation are performed respectively, and the event type detection result and the event start and end time detection result are outputted correspondingly, including: The audio-visual multimodal fusion encoding result, the video cross-attention special features and the audio cross-attention special features are input into the decoder, and classification calculation and regression calculation are performed respectively, and the event type detection result of the target audio and video file and the event start and end time detection result of the target audio and video file are correspondingly output.

5. The method for locating audio and video events according to claim 1, characterized in that: A cross-attention operation is performed according to the video attention feature and the audio feature to obtain a video cross-attention feature and an audio cross-attention feature, which are calculated by the following formula: ; ; in, Represents video features, Represents audio features, The cross-attention operation formula from audio modality to video modality is represented by: represents the query parameter matrix from the learnable audio modality to the video modality, represents the learnable key parameter matrix from the audio modality to the video modality, represents the learnable value parameter matrix from the audio modality to the video modality, Indicates the size of the audio feature, Indicates the duration of the complete video. The cross-attention operation formula from video modality to audio modality is represented by: represents the query parameter matrix from the learnable video modality to the audio modality, represents the key parameter matrix of the learnable video modality to audio modality, represents the learnable value parameter matrix from the video modality to the audio modality, Indicates the size of the video feature, Represents a normalization function, which is used to set the input number interval to between (-1,1).

6. The method for locating audio and video events according to claim 1, characterized in that: The audio-visual multimodal fusion encoding result is input into a decoder, and classification calculation and regression calculation are performed respectively, including: Classification and regression calculations are performed using the following formula: ; ; in, Indicates the time type with the highest probability of occurrence at each moment; Represents the learnable classification weight, which is , represents the number of video features, Indicates the number of candidate event types; represents the audio-visual multimodal fusion encoding result; Represents the index of the maximum value of the first dimension of the search vector; Represents the boundary regression parameter of the predicted time at each moment, and its size is , represents the moment, 2 represents two parameters that need to be regressed, one is It represents the ratio of the distance from the center of the event type identified at the current moment to the total video length. Indicates the ratio of the duration of the current event type to the unit duration of the current moment; represents the learnable regression weight; Represents a normalization function, which is used to set the input number interval to between (-1,1).

7. An audio and video event locating device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the audio and video event positioning method according to any one of claims 1 to 6 are implemented.

8. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the audio and video event locating method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the audio and video event locating method as claimed in any one of claims 1 to 6 are implemented.