Video analysis label optimization method, device and equipment based on weak supervised learning

Abstract

The invention particularly relates to a video parsing label optimization method, device and equipment based on weak supervised learning, and the method comprises the steps: carrying out the data preprocessing of a to-be-parsed video in a target data set, obtaining the audio features and visual features of the to-be-parsed video, carrying out the analysis after the feature aggregation, and obtaining a video parsing label; performing preliminary event prediction on the to-be-analyzed video according to an analysis result, obtaining an audio track of a first target video clip and a visual track of a second target video clip in the target data set, exchanging the audio track and the visual track to obtain a new to-be-analyzed video, and performing feature aggregation on the new to-be-analyzed video to obtain the to-be-analyzed video. Performing event prediction on a new to-be-analyzed video according to an analysis result, and respectively generating a first modal perception event tag and a second modal perception event tag according to the preliminary event prediction and the new event prediction, and optimizing the first modal perception event tag and the second modal perception event tag according to a comparison result of the preliminary event prediction and the new event prediction.

Description

Technical Field

[0001] This application relates to the technical field of audio-visual video parsing, and particularly relates to a method, device, and equipment for optimizing video parsing labels based on weakly supervised learning. Background Art

[0002] Audio-visual video parsing aims to conduct a detailed analysis of auditory and visual events in a video, and can parse an unconstrained video into a set of events, including categories, boundaries, and patterns. In deep learning, weakly supervised learning refers to a machine learning task where only partial annotation information is available for training a model, and the model is trained by leveraging the partial annotation information and other prior knowledge, thereby reducing the cost of data collection and annotation and can be applied to many practical scenarios.

[0003] When exploring and perceiving the sound environment through sensory streams, vision and audio are two basic streams that widely transmit a large amount of information in daily life. Compared with these visual models, audio-visual understanding is more robust in identifying ongoing events. For example, occlusion and blind spots are common in egocentric videos and online videos, where the object of interest is outside the field of view. In this case, auditory signals can provide reliable clues for video understanding, so it is necessary to use audio-visual models to parse videos.

[0004] In related technologies, audio-visual video parsing methods usually assume that audio and video data are always correlated and temporally aligned.

[0005] However, this consistency does not always hold in practical applications. It may be found that the sound of many videos comes from the scene view. At the same time, when related technologies handle weakly supervised learning tasks, they often rely on vague overall labels, resulting in poor model training effects. In addition, related technologies also have deficiencies in time localization ability and are difficult to accurately capture the time boundaries of events, which urgently need to be solved. Summary of the Invention

[0006] This application provides a method, device, and equipment for optimizing video parsing labels based on weakly supervised learning to solve problems such as poor model training effects caused by relying on vague overall labels in related technologies and difficulty in accurately capturing the time boundaries of events in time localization ability.

[0007] The first aspect embodiment of this application provides a method for optimizing video parsing labels based on weakly supervised learning, including the following steps:

[0008] Obtain a target data set, and perform data preprocessing on the video to be parsed in the target data set to obtain the audio features and visual features of the video to be parsed;

[0009] Using a preset attention mechanism to perform feature aggregation on the audio features and the visual features to obtain the aggregated features of the video to be parsed, and analyzing the aggregated features through a multi-instance learning mechanism, and performing preliminary event prediction on the video to be parsed according to the aggregated feature analysis result to obtain the preliminary video prediction result of the video to be parsed;

[0010] Obtain the audio track of the first target video segment in the target dataset and the visual track of the second target video segment in the target dataset, and exchange the audio track and the visual track to obtain a new video to be parsed, and perform feature aggregation on the new video to be parsed to obtain the aggregated features of the new video to be parsed, and perform event prediction on the new video to be parsed according to the new aggregated feature analysis result to obtain the new video prediction result of the new video to be parsed;

[0011] Generate the first-modal perception event label of the video to be parsed according to the preliminary video prediction result and generate the second-modal perception event label of the new video to be parsed according to the new video prediction result, and compare the preliminary video prediction result and the new video prediction result, and optimize the first-modal perception event label and the second-modal perception event label according to the comparison result.

[0012] According to an embodiment of the present application, the data preprocessing of the video to be parsed in the target dataset to obtain the audio features and visual features of the video to be parsed includes:

[0013] Segment the video to be parsed with a preset length;

[0014] Use a preset audio depth model to extract the audio features of each video segment, and use a preset visual depth model to extract the visual features of each video segment.

[0015] According to an embodiment of the present application, the using a preset attention mechanism to perform feature aggregation on the audio features and the visual features includes:

[0016] Use a preset self-attention mechanism to identify the first feature relationship between the audio features of each video segment, or the second feature relationship between the visual features of each video segment;

[0017] Use a preset cross-attention mechanism to identify the third feature relationship between the audio features and visual features of each video segment.

[0018] According to an embodiment of the present application, the performing preliminary event prediction on the video to be parsed according to the aggregated feature analysis result to obtain the preliminary video prediction result of the video to be parsed includes:

[0019] Performing event classification on each video segment in the video to be parsed according to the aggregation feature analysis result to obtain the probability of each event category in each video segment, so as to perform preliminary event prediction on each video segment according to the probability of each event category in each video segment;

[0020] Using the multi-instance learning mechanism to perform video segment aggregation according to the preliminary event prediction of each video segment to obtain the preliminary video prediction result of the video to be parsed.

[0021] According to an embodiment of the present application, comparing the preliminary video prediction result and the new video prediction result to optimize the first modality-aware event label and the second modality-aware event label according to the comparison result includes:

[0022] Determining a first confidence level of the video to be parsed according to the preliminary video prediction result, and determining a second confidence level of the new video to be parsed according to the new video prediction result;

[0023] Judging whether the first confidence level is lower than a preset threshold, or whether the second confidence level is lower than the preset threshold;

[0024] If the first confidence level is lower than the preset threshold, deleting the first modality-aware event label corresponding to the first confidence level, and if the second confidence level is lower than the preset threshold, deleting the second modality-aware event label corresponding to the second confidence level.

[0025] According to the video parsing label optimization method based on weakly supervised learning in the embodiment of the present application, performing data preprocessing on the video to be parsed in the target dataset to obtain the audio feature and visual feature of the video to be parsed, performing analysis after feature aggregation, so as to perform preliminary event prediction on the video to be parsed according to the analysis result, obtaining the audio track of the first target video segment and the visual track of the second target video segment in the target dataset, and swapping the audio track and the visual track to obtain a new video to be parsed and performing feature aggregation, then performing event prediction on the new video to be parsed according to the analysis result, generating a first modality-aware event label and a second modality-aware event label respectively according to the preliminary event prediction and the new event prediction, and optimizing the first modality-aware event label and the second modality-aware event label according to the comparison result of the preliminary event prediction and the new event prediction. Thus, the problems in the related art that the model training effect is not good due to relying on fuzzy overall labels and it is difficult to accurately capture the time boundary of events in terms of time localization ability are solved. By swapping the audio and video tracks to generate modality-aware event labels and combining cross-modal contrast learning to introduce time constraints to improve the time localization ability of the attention model.

[0026] The second aspect of the present application provides a video parsing label optimization device based on weakly supervised learning, including:

[0027] A data processing module, configured to obtain a target data set, and perform data preprocessing on the videos to be parsed in the target data set to obtain the audio features and visual features of the videos to be parsed;

[0028] A first prediction module, configured to use a preset attention mechanism to perform feature aggregation on the audio features and the visual features to obtain the aggregated features of the videos to be parsed, and analyze the aggregated features through a multi-instance learning mechanism, and perform preliminary event prediction on the videos to be parsed according to the aggregated feature analysis results to obtain the preliminary video prediction results of the videos to be parsed;

[0029] A second prediction module, configured to obtain the audio track of the first target video segment in the target data set and the visual track of the second target video segment in the target data set, exchange the audio track and the visual track to obtain a new video to be parsed, and perform feature aggregation on the new video to be parsed to obtain the aggregated features of the new video to be parsed, and perform event prediction on the new video to be parsed according to the new aggregated feature analysis results to obtain the new video prediction results of the new video to be parsed;

[0030] An optimization module, configured to generate the first-modal perception event labels of the videos to be parsed according to the preliminary video prediction results and generate the second-modal perception event labels of the new videos to be parsed according to the new video prediction results, and compare the preliminary video prediction results and the new video prediction results, and optimize the first-modal perception event labels and the second-modal perception event labels according to the comparison results.

[0031] According to an embodiment of the present application, the data processing module includes:

[0032] A segmentation processing unit, configured to segment the videos to be parsed with a preset length;

[0033] An extraction unit, configured to extract the audio features of each video segment by using a preset audio depth model, and extract the visual features of each video segment by using a preset visual depth model.

[0034] According to an embodiment of the present application, the first prediction module includes:

[0035] A first recognition unit, configured to use a preset self-attention mechanism to recognize the first feature relationship between the audio features of each video segment, or the second feature relationship between the visual features of each video segment;

[0036] A second recognition unit, configured to recognize a third feature relationship between the audio feature and the visual feature of each video segment by using a preset cross-attention mechanism.

[0037] According to an embodiment of the present application, the first prediction module includes:

[0038] Classify each video segment in the video to be parsed according to the aggregated feature analysis result to obtain the probability of each event category in each video segment, so as to perform a preliminary event prediction on each video segment according to the probability of each event category in each video segment;

[0039] Use the multi-instance learning mechanism to aggregate video segments according to the preliminary event prediction of each video segment to obtain a preliminary video prediction result of the video to be parsed.

[0040] According to an embodiment of the present application, the optimization module includes:

[0041] Determine a first confidence level of the video to be parsed according to the preliminary video prediction result, and determine a second confidence level of the new video to be parsed according to the new video prediction result;

[0042] Judge whether the first confidence level is lower than a preset threshold, or whether the second confidence level is lower than the preset threshold;

[0043] If the first confidence level is lower than the preset threshold, delete the first modality perception event label corresponding to the first confidence level. If the second confidence level is lower than the preset threshold, delete the second modality perception event label corresponding to the second confidence level.

[0044] The video parsing label optimization device based on weak supervision learning according to the embodiments of the present application preprocesses the video to be parsed in the target dataset to obtain the audio features and visual features of the video to be parsed. After feature aggregation, analysis is performed to perform preliminary event prediction on the video to be parsed according to the analysis results, obtain the audio track of the first target video segment and the visual track of the second target video segment in the target dataset, and exchange the audio track and the visual track to obtain a new video to be parsed. After feature aggregation, event prediction is performed on the new video to be parsed according to the analysis results. The first-modal perception event label and the second-modal perception event label are generated according to the preliminary event prediction and the new event prediction respectively, and the first-modal perception event label and the second-modal perception event label are optimized according to the comparison results of the preliminary event prediction and the new event prediction. Thus, the problems in the related art that the model training effect is not good due to relying on fuzzy overall labels and it is difficult to accurately capture the time boundary of events in terms of time localization ability are solved. The modal perception event label is generated by exchanging the audio and video tracks, and cross-modal contrast learning is combined to introduce time constraints to improve the time localization ability of the attention model.

[0045] An embodiment of the third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the program to implement the video parsing label optimization method based on weak supervision learning as described in the above embodiments.

[0046] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium. The computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the video parsing label optimization method based on weak supervision learning as described in the above embodiments.

[0047] An embodiment of the fifth aspect of the present application provides a computer program product, including a computer program / instruction, which when executed by a processor implements the video parsing label optimization method based on weak supervision learning as described in the above embodiments.

[0048] The additional aspects and advantages of the present application will be partly given in the following description, partly will become obvious from the following description, or will be understood through the practice of the present application. Description of the Drawings

[0049] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0050] Figure 1 It is a flowchart of a video parsing label optimization method based on weak supervision learning according to an embodiment of the present application;

[0051] Figure 2 is the overall implementation flowchart according to an embodiment of the present application;

[0052] Figure 3 is an example diagram of a video parsing label optimization device based on weakly supervised learning according to an embodiment of the present application;

[0053] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0054] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as a limitation to the present application.

[0055] The method, device and equipment for optimizing video parsing labels based on weakly supervised learning according to the embodiments of the present application will be described below with reference to the accompanying drawings. In view of the problems in the related art mentioned in the above background art, that is, the poor model training effect due to relying on fuzzy overall labels and the difficulty in accurately capturing the time boundaries of events in time localization ability, the present application provides a method for optimizing video parsing labels based on weakly supervised learning. In this method, the video to be parsed in the target dataset is preprocessed to obtain the audio features and visual features of the video to be parsed. After feature aggregation, analysis is performed to preliminarily predict events for the video to be parsed according to the analysis results, obtain the audio track of the first target video segment and the visual track of the second target video segment in the target dataset, and exchange the audio track and the visual track to obtain a new video to be parsed. After feature aggregation, event prediction is performed on the new video to be parsed according to the analysis results. The first modality-aware event label and the second modality-aware event label are generated respectively according to the preliminary event prediction and the new event prediction, and the first modality-aware event label and the second modality-aware event label are optimized according to the comparison results of the preliminary event prediction and the new event prediction. Thus, the problems in the related art, such as the poor model training effect due to relying on fuzzy overall labels and the difficulty in accurately capturing the time boundaries of events in time localization ability, are solved. Modality-aware event labels are generated by exchanging audio and video tracks, and cross-modal contrast learning is combined to introduce time constraints to improve the time localization ability of the attention model.

[0056] Specifically, before introducing the embodiments of the present application, the related technologies of the present application, as well as the methods and achieved purpose effects adopted by the present application, will be introduced first.

[0057] In deep learning, weakly supervised learning refers to a machine learning task where only partial annotation information is available for training a model, and this annotation information is usually incomplete or inaccurate. Different from traditional supervised learning, weakly supervised learning does not require a fully annotated dataset, but instead trains the model by leveraging partial annotated data and other prior knowledge. This method can reduce the cost of data collection and annotation and can be applied to many practical scenarios. Weakly supervised learning has been widely applied in fields such as computer vision, natural language processing, and audio processing. In the embodiments of this application, the focus is on the weakly supervised audio-visual video parsing task, whose purpose is to conduct a detailed analysis of auditory, visual, and audio-visual events in a video without assuming alignment.

[0058] Humans explore and perceive the sound environment through sensory streams, including vision, hearing, touch, etc. Among these co-existing sensory streams, vision and audio are two basic streams that widely transmit a large amount of information in daily life. Compared with these visual models, audio-visual understanding is more robust in identifying ongoing events. For example, occlusions and blind spots are common in ego-centric videos and online videos, where the object of interest is out of the field of view. In such cases, the auditory signal can provide reliable clues for video understanding. Therefore, it is necessary to use an audio-visual model to parse the video.

[0059] Existing audio-visual research works usually assume that audio and video data are always correlated and temporally aligned. However, this consistency may not always hold in practice. It may be found that the sound of many videos originates from the scene view. Although not aligned, the audio signal is still important for understanding these events, such as a motorcycle race outside the screen. In the embodiments of this application, the focus will be on the audio-visual video parsing task, aiming to provide a detailed analysis of auditory, visual, and audio-visual events in the video without such an alignment assumption.

[0060] To achieve this goal, the embodiments of this application will swap the audio and video tracks of the training video with other unrelated videos. The motivation is that if the visual or audio signal does contain clues to the target event, the prediction of the newly assembled video will still be very confident. Otherwise, the event information is invisible or inaudible in the corresponding modality. Thus, precise modality-aware event labels can be further obtained and the model can be protected from being misled by ambiguous overall labels.

[0061] Specifically, Figure 1 is a schematic flowchart of a video parsing label optimization method based on weakly supervised learning provided by the embodiments of this application.

[0062] As Figure 1 shown, the video parsing label optimization method based on weakly supervised learning includes the following steps:

[0063] In step S101, a target data set is obtained, and data preprocessing is performed on the video to be parsed in the target data set to obtain the audio features and visual features of the video to be parsed.

[0064] According to an embodiment of the present application, data preprocessing is performed on the video to be parsed in the target data set to obtain the audio features and visual features of the video to be parsed, including: segmenting the video to be parsed at a preset length; using a preset audio depth model to extract the audio features of each video segment, and using a preset visual depth model to extract the visual features of each video segment.

[0065] Specifically, to solve the problem that the overall event label damages model training due to audiovisual asynchrony, the embodiment of the present application exchanges the audio track and the video track with an unrelated video to accurately supervise the events perceived by the modality. In addition, the embodiment of the present application combines the cross-modal audiovisual contrast learning method, introduces temporal constraints into the attention model, and alleviates the problem of blurred temporal boundaries in the weakly supervised AVVP (Audio-Visual Video Parsing) task.

[0066] Among them, the preset length, the preset audio depth model, and the preset visual depth model can be selected by those skilled in the art according to actual usage requirements, and no specific limitation is made here.

[0067] Specifically, as Figure 2 shown, first, a video data set containing audio is obtained from open source channels (such as GitHub, Kaggle, academic papers, etc.), such as the LLP data set (Look, Listen, and Parse data set). This target data set is the basis for subsequent model training; secondly, data preprocessing needs to be performed on the video to be parsed in the target data set to convert the original data into a feature representation available for the model, including segmenting the video to be parsed at a preset length. Usually, the video to be parsed is divided into several fixed-length segments (such as 1 second per segment), and a preset audio depth model (such as the VGGish network model, which is a model improved according to VGG (Visual Geometry Group)) is used to extract the audio features of each video segment and a preset visual depth model (such as the ResNet-152 and R(2+1)D models) is used to extract the visual features of each video segment The extracted audio features and visual features are used as the input for the following feature aggregation and multi-instance learning to ensure that the audio features and video features are temporally aligned, facilitating subsequent multimodal fusion.

[0068] Among them, the target dataset of the embodiments of the present application contains 11,849 video clips and 25 event categories, covering a wide range of daily life scenarios, including human activities, animal activities, music performances, and vehicle sounds. The detailed event category annotations, including men speaking, dogs barking, playing the guitar, and frying food, etc., last for 10 seconds, including audio and video tracks. There are 7,202 videos containing events from multiple event categories, and each video has an average of 1.64 different event categories. For the weakly supervised AVVP task, 10,000 videos are used for training, only containing weak labels (video-level event annotations on whether different video events exist). To evaluate the AVVP performance, 1,849 validation and test videos have fully annotated labels, that is, separate audio and video events with time boundaries.

[0069] In step S102, a preset attention mechanism is used to perform feature aggregation on the audio features and visual features to obtain the aggregated features of the video to be parsed, and a multi-instance learning mechanism is used to analyze the aggregated features. According to the analysis result of the aggregated features, a preliminary event prediction is made for the video to be parsed, and a preliminary video prediction result of the video to be parsed is obtained.

[0070] According to an embodiment of the present application, using a preset attention mechanism to perform feature aggregation on the audio features and visual features includes: using a preset self-attention mechanism to identify the first feature relationship between the audio features of each video clip, or the second feature relationship between the visual features of each video clip; using a preset cross-attention mechanism to identify the third feature relationship between the audio features and visual features of each video clip.

[0071] Among them, the preset attention mechanism may include a preset self-attention mechanism and a preset cross-attention mechanism. The preset self-attention mechanism and the preset cross-attention mechanism can both be selected by those skilled in the art according to actual usage requirements, and no specific limitation is made here.

[0072] Specifically, as Figure 2 shown, after the audio features and visual features are extracted in the embodiments of the present application, it is necessary to further use a preset attention mechanism to perform feature aggregation on the audio features and visual features, so as to use context information to enhance the input features, enhance the expression ability of the features, and capture the complex relationships within and between modalities.

[0073] Specifically, in the embodiments of the present application, a preset self-attention mechanism can be used to capture the feature relationships within the same modality (such as the relationships between visual features or audio features), that is, the preset self-attention mechanism is used to identify the first feature relationship between the audio features of each video segment or the second feature relationship between the visual features of each video segment. In addition, a preset cross-attention mechanism can also be used to capture the feature relationships between different modalities (such as the relationship between visual features and audio features), that is, to identify the third feature relationship between the audio features and visual features of each video segment. Thus, compared with the audio features and visual features before the above enhancement, the audio aggregation features are improved by collecting the event information in the entire video segment and visual aggregation features quality.

[0074] The specific implementation method is as follows. att(·) is the scaled dot product performed on the query, key, and value.

[0075]

[0076] where d is the dimension of the feature vector q, K is the key, T is the matrix transpose, V is the value, and the aggregation feature can be obtained through

[0077] the following method:

[0078]

[0079]

[0080] where, is the audio feature sequence from the same video, is the video feature sequence from the same video.

[0081] According to an embodiment of the present application, based on the aggregation feature analysis result, a preliminary event prediction is performed on the video to be parsed, and a preliminary video prediction result of the video to be parsed is obtained, including: classifying events for each video segment in the video to be parsed according to the aggregation feature analysis result to obtain the probability of each event category in each video segment, so as to perform a preliminary event prediction on each video segment according to the probability of each event category in each video segment; using a multi-instance learning mechanism to aggregate video segments according to the preliminary event prediction of each video segment to obtain a preliminary video prediction result of the video to be parsed.

[0082] Specifically, after feature aggregation of the audio features and visual features, a multi-instance learning mechanism can be used to analyze the aggregation features, so as to perform a preliminary event prediction on the video to be parsed according to the aggregation feature analysis result, and obtain a preliminary video prediction result of the video to be parsed.

[0083] Specifically, as Figure 2 shown, since the event prediction for each video segment is based on the aggregated audio and visual features, and multiple events may occur in the same video segment, therefore, in the embodiments of the present application, the Sigmoid function can be used on the classifier to output the probability of each event category, and and can be respectively represented as the event prediction for the audio and visual features of the t-th segment. At the same time, since the present application adopts weak supervision learning and can only access the weak labels at the video level instead of the accurate video segment-level labels, therefore, it is necessary to further use the attention multi-instance pooling method to predict the event probability at the video level. The recognition of event categories usually requires combining the information of audio and video segments and is achieved through multimodal fusion. In the case of only having video-level labels, the model can infer the specific segments where events occur.

[0084] Specifically, first, event classification is performed on each video segment in the video to be parsed according to the aggregated feature analysis results. For example, each video segment can be classified into user speech, car horn sound, etc., and then the model outputs the probability of each event category in each video segment. For example, the probability of user speech in video segment 1 is 0.8, and the probability of car horn sound is 0.2, or the probability of user speech in video segment 2 is 0.3, and the probability of car horn sound is 0.7, etc. After obtaining the probability of each event category in each video segment, preliminary event prediction is performed on each video segment according to the probability of each event category in each video segment, and then the multi-instance learning mechanism is used to aggregate the video segments according to the preliminary event prediction of each video segment to obtain the preliminary video prediction result of the video to be parsed, that is to say, the preliminary event prediction at the video segment level can be aggregated through multi-instance learning to obtain the preliminary video prediction result of the video to be parsed at the video level. For example, if the prediction probability of an event is high in multiple segments, the model considers that the entire video contains this event.

[0085] It should be noted that the video-level audio event probability and the visual event probability are obtained through the weighted average of all segment-level predictions. For the required baseline, the binary cross-entropy loss function can be used to optimize the video-level probabilities and to approach the overall event label

[0086] In step S103, obtain the audio track of the first target video segment in the target dataset and the visual track of the second target video segment in the target dataset, swap the audio track and the visual track, obtain a new video to be parsed, perform feature aggregation on the new video to be parsed to obtain the aggregated features of the new video to be parsed, and perform event prediction on the new video to be parsed according to the analysis result of the new aggregated features to obtain the new video prediction result of the new video to be parsed.

[0087] Specifically, as Figure 2 shown, in the embodiment of the present application, the modality uncertainty problem is alleviated by swapping the audio track and the video track with other video segments. If the prediction confidence of the newly combined video for the target event is still very high, it indicates that the original modality (audio or video) contains clues to the target event. Otherwise, it indicates that the original modality does not contain clues to the target event. Thus, by generating adversarial samples, the contribution of each modality to the target event can be evaluated, providing a basis for subsequent modality-aware event label generation.

[0088] Specifically, first, obtain the audio track of the first target video segment in the target dataset and the visual track of the second target video segment in the target dataset, swap the audio track and the visual track to obtain a new video to be parsed; second, perform feature aggregation on the new video to be parsed, obtain the aggregated features of the new video to be parsed based on the above feature aggregation step, and perform event prediction on the new video to be parsed according to the analysis result of the new aggregated features to obtain the new video prediction result of the new video to be parsed; third, if the visual features and audio features of the original video do contain clues to the target event, the prediction of the new video to be parsed still has a very high confidence, and it is necessary to obtain the video prediction result of the new video to be parsed to evaluate the uncertainty.

[0089] For example, assume there are two video segments with non-overlapping video-level event labels, i.e., S i =(V i , A i ) and S j =(V j , A j ), but take the video S i =(V i , A i ) as the target video, swap the visual track and the audio track of these two videos, and form a new "video" in the following way:

[0090]

[0091] Among them, represents being from the video segment S iThe visual features and the audio features from video clip S j to form a new "video", due to the video-level event label ensures that there is no event j in any modality of video S Therefore, it can be concluded that V j and A j are both irrelevant to the target event Therefore, for the newly combined data and the only clue regarding event information comes from the content of the i-th video S i i.e., from V i , A i or both.

[0092] Furthermore, the embodiments of the present application represent the basic model as obtain event predictions for these assembled videos in the following manner:

[0093]

[0094] wherein, is the event prediction based on the aggregated visual features of the video with changed audio, is the event prediction based on the aggregated visual features of the video with changed visual content, is the event prediction based on the aggregated audio features of the video with changed audio, is the event prediction based on the aggregated audio features of the video with changed visual content, E c is the normalized error rate intuition obtained according to the target event category C predicted by training. If it is found difficult to optimize the corresponding event category, the situation of label misalignment is more likely to occur (the training accuracy of event category C is low). From this, it can be shown through prediction and the reliability of the event labels of the visual track in video S i Similarly, and are used to verify the reliability of the event labels of the audio track.

[0095] In step S104, generate the first-modal perception event labels of the video to be parsed according to the preliminary video prediction results and generate the second-modal perception event labels of the new video to be parsed according to the new video prediction results, and compare the preliminary video prediction results with the new video prediction results, and optimize the first-modal perception event labels and the second-modal perception event labels according to the comparison results.

[0096] According to an embodiment of the present application, the preliminary video prediction result and the new video prediction result are compared to optimize the first-modal perception event label and the second-modal perception event label according to the comparison result, including: determining a first confidence level of the video to be parsed according to the preliminary video prediction result, and determining a second confidence level of the new video to be parsed according to the new video prediction result; determining whether the first confidence level is lower than a preset threshold or whether the second confidence level is lower than the preset threshold; if the first confidence level is lower than the preset threshold, deleting the first-modal perception event label corresponding to the first confidence level, and if the second confidence level is lower than the preset threshold, deleting the second-modal perception event label corresponding to the second confidence level.

[0097] Wherein, the preset threshold can be selected by those skilled in the art according to actual usage requirements, and no specific limitation is made here.

[0098] Specifically, as Figure 2 shown, the embodiment of the present application further refines the event label by further evaluating the confidence level of each modality and provides different event labels for these two modalities.

[0099] Specifically, first, the embodiment of the present application generates a modality-perception event label according to the confidence level of the prediction result, that is, generates a first-modal perception event label of the video to be parsed according to the preliminary video prediction result and generates a second-modal perception event label of the new video to be parsed according to the new video prediction result; secondly, determines a first confidence level of the video to be parsed according to the preliminary video prediction result and determines a second confidence level of the new video to be parsed according to the new video prediction result. If the confidence level of a certain modality is lower than the preset threshold (for example, 0.5), it is considered that this modality does not contain clues of the target event, and the event label of this modality is deleted. For example, if the first confidence level is lower than the preset threshold, the first-modal perception event label corresponding to the first confidence level is deleted, and if the second confidence level is lower than the preset threshold, the second-modal perception event label corresponding to the second confidence level is deleted, so as to ensure the accuracy and independence of the event label of each modality by deleting the event label with low confidence level. That is to say, if the confidence level of the visual modality is lower than the preset threshold, the event label of the visual modality is deleted, and only the label of the audio modality is retained.

[0100] For example, if the confidence level is lower than the preset threshold of 0.5, the event label is reallocated and the irrelevant labels of each modality are deleted, because the range of sigmoid prediction is [0, 1]. Specifically, if and the event label of the visual modality will be discarded. Similarly, if and the event label of the audio modality is deleted. Thus, it can be roughly estimated whether the event occurs on the visual track or the auditory track through the modality-perception label.

[0101] Furthermore, in the embodiments of the present application, contrastive learning is used to further induce temporal differences in the attention model, thereby obtaining better temporal localization performance. First, contrastive learning is introduced, and noise contrast estimation is used to enhance the temporal localization ability. A set of candidate distractors is constructed, which contains the visual features of the same video but in different time periods; the cosine similarity between the preliminary video prediction result and the new video prediction result is calculated. Through contrastive learning, the model is encouraged to establish stronger associations between features in close time periods and move away from features in other time periods. Thus, through contrastive learning, the temporal localization ability of the model can be optimized, enabling it to more accurately capture the temporal boundaries of events.

[0102] Specifically, noise contrast estimation is used to encourage the aggregation of features of low-level visual features in close time periods and move away from visual features in other time periods. Therefore, the only target required is the true feature Then, a set of candidates is constructed as distractors, which contains the visual features of the same video but in different time periods, that is where t’≠t. Since these candidates are difficult to distinguish because they are very close to the true frame features Therefore, with the target and these distractors, auxiliary supervision can be added to the model through contrastive learning. First, the cosine similarity between the predicted features and the candidate features is calculated At this time, all vectors are forced to be L2-normalized feature embeddings, that is, There is the following objective function within the time period t:

[0103]

[0104] where, is the visual feature at the j-th second, τ is the temperature parameter that controls the concentration level of the distribution, T is the temperature coefficient that controls the concentration level of the distribution, a higher τ results in a softer probability distribution. In the experiment, τ = 0.2 is set. By combining the binary cross-entropy loss and the above contrastive loss, the attention model may not be dominated by certain time periods, and the aggregated features are more likely to be the information that occurs in this segment rather than all context features, thereby obtaining better temporal localization performance.

[0105] In summary, the embodiments of the present application evaluate the method adopted in the present application by parsing all types of events (audio, video, and audiovisual events) under segment-level and event-level metrics. The F1 score (a metric used to evaluate the performance of a classification model, which comprehensively considers precision and recall and measures the performance of the model through the harmonic mean of the two) is used as the metric for evaluating predictions. The segment-level metrics evaluate the performance of segment event predictions. In addition to segment-level performance, event-level results are also reported to indicate the performance in actual applications. To calculate the event-level F1 score results, events are extracted by concatenating consecutive valid segments in the same event category, and mIoU = 0.5 is used as the threshold for calculating the event-level F1 score. In addition, the overall audiovisual scene parsing performance of the method is evaluated by calculating the summary results, namely "Type@AV" and "Event@AV", as shown in Table 1. Specifically, Type@AV calculates the average evaluation results of audio, video, and audiovisual events, while Event@AV calculates the F1 score by considering all audio and video event types of each sample, rather than directly averaging the results of different event types.

[0106] Table 1

[0107]

[0108] Specifically, the embodiments of the present application use the same visual features and audio features as previous works for fair comparison. The ResNet-152 model pre-trained on ImageNet and the 18-layer deep R(2+1)D model pre-trained on Kinetics-400 are used to extract the visual features of each video segment. The video is decoded at 8fps and each segment (lasting one second) is input to obtain 2D and 3D visual features, and the concatenation of the two visual features is regarded as the low-level visual feature. For the audio signal, the VGGish network pre-trained on AudioSet is used to extract the audio features of each video segment, and the Adam optimizer is used to train the framework. The mini-batch size is 16 and the learning rate is 3×10-4. In the embodiments of the present application, it can be trained for 40 epochs, and the learning rate is reduced by 10 times after 10 epochs. This training method includes three stages: First, use MIL (Multi-Instance Learning) and the proposed contrastive learning to optimize the basic model for audiovisual scene parsing; Second, freeze the model and evaluate each video by swapping its audio and video tracks with other irrelevant videos; Finally, re-train the model from scratch using modality-aware labels, and name the final model "MA" (Modality-Aware label refining) to distinguish it from the basic model.

[0109] Therefore, in the embodiments of the present application, the model MA is compared with the weakly supervised sound detection method TALNet, the temporal action localization method STPN (Sparse Temporal Pooling Network), and CMCS (Cross-Modal Correlation Sampling), as well as the state-of-the-art audio-visual event parsing methods, including AVE (Audio-Visual Embedding)

[40] , AVSDN (Audio-Visual Scene Description Network), and HAN (Hierarchical Attention Network). All models (including the models used in the embodiments of the present application) are only trained using the LLP training dataset for fair comparison, including the same training data and preprocessed audio / video features. As can be seen from Table 1 above, the performance of the MA method in the embodiments of the present application and the state-of-the-art methods on the LLP test set is shown, and it can be seen that the method adopted in the embodiments of the present application is significantly superior to the state-of-the-art methods in all audio-visual video parsing subtasks at the segment level and event level.

[0110] Specifically, regarding the prediction of audio-visual events, MA in the embodiments of the present application exceeded the state-of-the-art method HAN by 6.2% at the segment level (from 48.9% to 55.1%) and by 6.0% at the event level (from 43.0% to 49.0%). The most significant improvement is in visual event parsing, which verifies the motivation of the embodiments of the present application, that is, previous methods are affected by the overall labels of invisible events that are ambiguous. The comparison with the latest methods shows that the model in the embodiments of the present application can significantly better predict event categories through accurate temporal positions.

[0111] Furthermore, ablation experiments are conducted in the embodiments of the present application to show the effectiveness of modality-aware improvement. As shown in Table 2, "Baseline + R" represents the results of the model trained using modality-aware refinement. By leveraging the cues between the audio and video tracks and assigning different labels to the two modalities, it is found that the model performance is significantly improved. Table 2 shows that the model "Baseline + R" adopted in the embodiments of the present application is about 4% higher than the baseline in the audio-visual event parsing evaluation metrics. Specifically, for visual event parsing, the model with modality-aware refinement significantly improves the performance by 4.6% (from 52.9% to 57.5%) in segment-level predicates and by 5.5% (from 48.9% to 54.4%) at the event level, verifying that ambiguous video-level labels can harm model training because some events only appear in one modality.

[0112] Table 2

[0113]

[0114] Furthermore, Table 2 also shows the relative improvement brought by cross-modal contrastive learning. Compared with the baseline, the model of the embodiment of the present application that only uses contrastive learning ("Baseline + C") has improved in the parsing of the audiovisual model. When combined with modal perception improvement, the relative improvement is more significant. By comparing the model "Baseline + C + R" and the model "Baseline + R", it can be found that contrastive learning further improves the event parsing performance by about 2% on most evaluation metrics, which indicates that the contrastive learning proposed in the present application can introduce essential temporal differences for audiovisual video parsing.

[0115] Furthermore, the embodiment of the present application further reveals the effect of modal perception refinement by studying the modality. The experiments conducted in the embodiment of the present application include: (1) only refining audio labels; (2) only refining visual labels; (3) refining labels of both modalities. The experimental results are reported in Table 3. It can be found that the most significant improvement is in visual parsing prediction by optimizing event labels. By optimizing event labels, the performance of segment-level visual parsing evaluation is significantly improved. The reason is that visual content can only be provided by the camera, and whether the object of interest is usually out of the field of view or not. In contrast, the audio signal is collected by the microphone, and the microphone can perceive all event information of the scene. Therefore, mismatched event labels are more common in the visual modality. By optimizing the visual event labels of these audible but invisible videos, it can be observed that the performance of all evaluation metrics except pure audio parsing has been significantly improved.

[0116] In addition, the embodiment of the present application further improves the performance by optimizing the event labels of both modalities. Compared with "only visual objects", the model trained with the optimization of both modalities has obtained considerable performance improvement in all evaluation metrics.

[0117] Table 3

[0118]

[0119] Furthermore, in the obtained objective function L c above, τ is the temperature parameter that controls the concentration level of the distribution. The present application further verifies different τ values used in the experiment. As shown in Table 4, which shows the comparison of segment-level audiovisual video parsing evaluation, the smaller τ is, the clearer the probability distribution is. In the experiment, it can be found that as τ decreases, the performance will be slightly higher. Generally speaking, the model adopted in the embodiment of the present application is not sensitive to the τ value used in contrastive learning. In all other experiments, τ can be set to 0.2.

[0120] Table 4

[0121]

[0122] Thus, the embodiments of the present application design a method for accurately supervising the events perceived by the modality by swapping the audio track and the video track of an unrelated video, rather than only obtaining vague overall labels. The present application also uses the method of contrastive learning to introduce time constraints into the attention model, alleviating the problem of blurred time boundaries in the weakly supervised AVVP task.

[0123] In summary, based on the specific discussions of the above embodiments, the present application can achieve the following beneficial effects:

[0124] (1) By swapping the audio track and the video track of an unrelated video, we can accurately supervise the events perceived by the modality, rather than only obtaining vague overall labels;

[0125] (2) Using the method of contrastive learning, we introduce time constraints into the attention model, alleviating the problem of blurred time boundaries in the weakly supervised AVVP task;

[0126] (3) Based on the above evaluation metrics, the segment-level audiovisual parsing accuracy of the LLP dataset has been improved from 48.9% to 55.1%.

[0127] According to the video parsing label optimization method based on weakly supervised learning of the embodiments of the present application, data preprocessing is performed on the video to be parsed in the target dataset to obtain the audio features and visual features of the video to be parsed. After feature aggregation, analysis is performed to perform preliminary event prediction on the video to be parsed according to the analysis results, obtain the audio track of the first target video segment and the visual track of the second target video segment in the target dataset, and swap the audio track and the visual track to obtain a new video to be parsed. After feature aggregation, event prediction is performed on the new video to be parsed according to the analysis results. The first modality perception event label and the second modality perception event label are generated respectively according to the preliminary event prediction and the new event prediction, and the first modality perception event label and the second modality perception event label are optimized according to the comparison result of the preliminary event prediction and the new event prediction. Thus, the problems in the related art, such as poor model training effect due to relying on vague overall labels and difficulty in accurately capturing the time boundaries of events in terms of time localization ability, are solved. Modality perception event labels are generated by swapping the audio and video tracks, and time constraints are introduced through cross-modal contrastive learning to improve the time localization ability of the attention model.

[0128] Next, a video parsing label optimization device based on weakly supervised learning proposed according to the embodiments of the present application will be described with reference to the accompanying drawings.

[0129] Figure 3 It is a block diagram of the video parsing label optimization device based on weakly supervised learning of the embodiments of the present application.

[0130] As Figure 3 shown, the video parsing label optimization device 10 based on weakly supervised learning includes: a data processing module 100, a first prediction module 200, a second prediction module 300, and an optimization module 400.

[0131] Among them, the data processing module 100 is used to obtain a target data set and perform data preprocessing on the video to be parsed in the target data set to obtain the audio features and visual features of the video to be parsed;

[0132] The first prediction module 200 is used to aggregate the audio features and visual features by using a preset attention mechanism to obtain the aggregated features of the video to be parsed, and analyze the aggregated features through a multi-instance learning mechanism, and perform preliminary event prediction on the video to be parsed according to the analysis result of the aggregated features to obtain the preliminary video prediction result of the video to be parsed;

[0133] The second prediction module 300 is used to obtain the audio track of the first target video segment in the target data set and the visual track of the second target video segment in the target data set, exchange the audio track and the visual track to obtain a new video to be parsed, and after aggregating the features of the new video to be parsed, obtain the aggregated features of the new video to be parsed, and perform event prediction on the new video to be parsed according to the analysis result of the new aggregated features to obtain the new video prediction result of the new video to be parsed;

[0134] The optimization module 400 is used to generate the first-modal perception event label of the video to be parsed according to the preliminary video prediction result and generate the second-modal perception event label of the new video to be parsed according to the new video prediction result, and compare the preliminary video prediction result and the new video prediction result, and optimize the first-modal perception event label and the second-modal perception event label according to the comparison result.

[0135] According to an embodiment of the present application, the data processing module 100 includes:

[0136] A segmentation processing unit for segmenting the video to be parsed with a preset length;

[0137] An extraction unit for extracting the audio features of each video segment by using a preset audio depth model and extracting the visual features of each video segment by using a preset visual depth model.

[0138] According to an embodiment of the present application, the first prediction module 200 includes:

[0139] A first recognition unit for identifying the first feature relationship between the audio features of each video segment or the second feature relationship between the visual features of each video segment by using a preset self-attention mechanism;

[0140] A second recognition unit, configured to recognize a third feature relationship between the audio feature and the visual feature of each video segment by using a preset cross-attention mechanism.

[0141] According to an embodiment of the present application, the first prediction module 200 includes:

[0142] Classify each video segment in the video to be parsed according to the aggregated feature analysis result, and obtain the probability of each event category in each video segment, so as to perform a preliminary event prediction on each video segment according to the probability of each event category in each video segment;

[0143] Use a multi-instance learning mechanism to aggregate video segments according to the preliminary event prediction of each video segment, and obtain a preliminary video prediction result of the video to be parsed.

[0144] According to an embodiment of the present application, the optimization module 400 includes:

[0145] Determine a first confidence level of the video to be parsed according to the preliminary video prediction result, and determine a second confidence level of the new video to be parsed according to the new video prediction result;

[0146] Judge whether the first confidence level is lower than a preset threshold, or whether the second confidence level is lower than the preset threshold;

[0147] If the first confidence level is lower than the preset threshold, delete the first modality perception event label corresponding to the first confidence level; if the second confidence level is lower than the preset threshold, delete the second modality perception event label corresponding to the second confidence level.

[0148] The video parsing label optimization device based on weak supervision learning according to the embodiments of the present application preprocesses the video to be parsed in the target dataset to obtain the audio features and visual features of the video to be parsed. After feature aggregation, analysis is performed to perform preliminary event prediction on the video to be parsed according to the analysis results, obtain the audio track of the first target video segment and the visual track of the second target video segment in the target dataset, and exchange the audio track and the visual track to obtain a new video to be parsed. After feature aggregation, event prediction is performed on the new video to be parsed according to the analysis results. The first modality perception event label and the second modality perception event label are generated respectively according to the preliminary event prediction and the new event prediction, and the first modality perception event label and the second modality perception event label are optimized according to the comparison results of the preliminary event prediction and the new event prediction. Thus, the problems in the related art that the model training effect is not good due to relying on fuzzy overall labels and it is difficult to accurately capture the time boundary of events in terms of time positioning ability are solved. The modality perception event label is generated by exchanging the audio and video tracks, and the time constraint is introduced through cross-modal contrast learning to improve the time positioning ability of the attention model.

[0149] Figure 4 The structure diagram of the electronic device provided by the embodiments of the present application is shown. The electronic device may include:

[0150] A memory 401, a processor 402, and a computer program stored on the memory 401 and executable on the processor 402.

[0151] When the processor 402 executes the program, it implements the video parsing label optimization method based on weak supervision learning provided in the above embodiments.

[0152] Further, the electronic device further includes:

[0153] A communication interface 403 for communication between the memory 401 and the processor 402.

[0154] The memory 401 is used to store a computer program executable on the processor 402.

[0155] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0156] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be interconnected through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 only a thick line is used in Figure 4 , but it does not mean that there is only one bus or one type of bus.

[0157] Optionally, in specific implementation, if the memory 401, the processor 402, and the communication interface 403 are integrated on a chip, the memory 401, the processor 402, and the communication interface 403 can communicate with each other through an internal interface.

[0158] The processor 402 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0159] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above method for optimizing video parsing tags based on weak supervision learning is implemented.

[0160] The embodiments of the present application further provide a computer program product, including computer programs / instructions, and when the computer programs / instructions are executed by a processor, the method for optimizing video parsing tags based on weak supervision learning as described in the above embodiments is implemented.

[0161] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc., mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0162] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0163] Any process or method description shown in a flowchart or described in other ways herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logic function or process. And the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the functions involved, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of this application pertain.

[0164] The logic and / or steps represented in the flowchart or otherwise described herein can, for example, be considered as a definable sequence list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) having one or N wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, a computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or, if necessary, other suitable processing, and then storing it in a computer memory.

[0165] It should be understood that the various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), and the like.

[0166] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried out in implementing the above-described example methods can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0167] In addition, each functional unit in various embodiments of the present application may be integrated into one processing module, may exist separately physically for each unit, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0168] The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present application.

Claims

1. A method for optimizing video parsing labels based on weakly supervised learning, characterized in that, Including the following steps: Obtain a target data set, and perform data preprocessing on the video to be parsed in the target data set to obtain the audio features and visual features of the video to be parsed; Use a preset attention mechanism to perform feature aggregation on the audio features and the visual features to obtain the aggregated features of the video to be parsed, and analyze the aggregated features through a multi-instance learning mechanism, and perform preliminary event prediction on the video to be parsed according to the aggregated feature analysis result to obtain the preliminary video prediction result of the video to be parsed; Obtain the audio track of the first target video segment in the target data set and the visual track of the second target video segment in the target data set, exchange the audio track and the visual track to obtain a new video to be parsed, and perform feature aggregation on the new video to be parsed to obtain the aggregated features of the new video to be parsed, and perform event prediction on the new video to be parsed according to the new aggregated feature analysis result to obtain the new video prediction result of the new video to be parsed; Generate the first-modal perception event label of the video to be parsed according to the preliminary video prediction result and generate the second-modal perception event label of the new video to be parsed according to the new video prediction result, and compare the preliminary video prediction result and the new video prediction result, and optimize the first-modal perception event label and the second-modal perception event label according to the comparison result.

2. The method according to claim 1, characterized in that The data preprocessing of the video to be parsed in the target data set to obtain the audio features and visual features of the video to be parsed includes: Segment the video to be parsed at a preset length; Use a preset audio depth model to extract the audio features of each video segment, and use a preset visual depth model to extract the visual features of each video segment.

3. The method according to claim 1 or 2, characterized in that, The using of the preset attention mechanism to perform feature aggregation on the audio features and the visual features includes: Use a preset self-attention mechanism to identify the first feature relationship between the audio features of each video segment, or the second feature relationship between the visual features of each video segment; Use a preset cross-attention mechanism to identify the third feature relationship between the audio features and the visual features of each video segment.

4. The method according to claim 1, wherein The performing of preliminary event prediction on the video to be parsed according to the aggregated feature analysis result to obtain the preliminary video prediction result of the video to be parsed includes: Classify events for each video segment in the video to be parsed according to the aggregated feature analysis result to obtain the probability of each event category in each video segment, so as to perform preliminary event prediction on each video segment according to the probability of each event category in each video segment; Use the multi-instance learning mechanism to perform video segment aggregation according to the preliminary event prediction of each video segment to obtain the preliminary video prediction result of the video to be parsed.

5. The method according to claim 1, characterized in that, The comparing of the preliminary video prediction result and the new video prediction result, and the optimizing of the first-modal perception event label and the second-modal perception event label according to the comparison result includes: Determine a first confidence level of the video to be parsed according to the preliminary video prediction result, and determine a second confidence level of the new video to be parsed according to the new video prediction result; Judge whether the first confidence level is lower than a preset threshold, or whether the second confidence level is lower than the preset threshold; If the first confidence level is lower than the preset threshold, delete the first modality perception event label corresponding to the first confidence level. If the second confidence level is lower than the preset threshold, delete the second modality perception event label corresponding to the second confidence level.

6. A video parsing label optimization device based on weak supervision learning, characterized in that, Comprising: A data processing module, configured to obtain a target data set, and perform data preprocessing on the video to be parsed in the target data set to obtain the audio feature and visual feature of the video to be parsed; A first prediction module, configured to use a preset attention mechanism to perform feature aggregation on the audio feature and the visual feature to obtain an aggregated feature of the video to be parsed, and analyze the aggregated feature through a multi-instance learning mechanism, and perform preliminary event prediction on the video to be parsed according to the aggregated feature analysis result to obtain a preliminary video prediction result of the video to be parsed; A second prediction module, configured to obtain the audio track of the first target video segment in the target data set and the visual track of the second target video segment in the target data set, exchange the audio track and the visual track to obtain a new video to be parsed, perform feature aggregation on the new video to be parsed, obtain the aggregated feature of the new video to be parsed, and perform event prediction on the new video to be parsed according to the new aggregated feature analysis result to obtain a new video prediction result of the new video to be parsed; An optimization module, configured to generate a first modality perception event label of the video to be parsed according to the preliminary video prediction result and generate a second modality perception event label of the new video to be parsed according to the new video prediction result, compare the preliminary video prediction result and the new video prediction result, and optimize the first modality perception event label and the second modality perception event label according to the comparison result.

7. The device according to claim 6, wherein The data processing module includes: A segmentation processing unit, configured to segment the video to be parsed with a preset length; An extraction unit, configured to extract the audio feature of each video segment by using a preset audio depth model and extract the visual feature of each video segment by using a preset visual depth model.

8. The device according to claim 6 or 7, characterized in that, The first prediction module includes: A first recognition unit, configured to use a preset self-attention mechanism to recognize a first feature relationship between the audio features of each video segment, or a second feature relationship between the visual features of each video segment; A second recognition unit, configured to use a preset cross-attention mechanism to recognize a third feature relationship between the audio feature and the visual feature of each video segment.

9. An electronic device, characterized in that, Comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the video parsing label optimization method based on weak supervision learning according to any one of claims 1-5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method for optimizing video parsing labels based on weakly supervised learning according to any one of claims 1-5.

Images