Dense video description method based on progressive modal disentanglement

Abstract

This invention provides a dense video description method based on progressive modal decoupling, involving data processing technology. It involves obtaining initial multimodal data for model training from training videos; performing temporal localization processing on the visual feature data of the initial multimodal data based on the temporal boundary data of the training videos to obtain multimodal data with enhanced geometric boundary awareness; designing modal awareness factors; and performing semantic boundary decoupling processing on the training videos based on the modal awareness factors and semantic transformation cues to obtain final multimodal feature data; and performing description generation processing on the final multimodal feature data based on a cross-attention autoregressive mechanism to generate event description data, and sorting the event description data according to chronological order to generate an event description sequence. This allows the model to utilize multiple types of information in a balanced manner during training, enhancing its event boundary awareness capability and improving the accuracy of generated video descriptions.

Description

Technical Field

[0001] This invention relates to data processing technology, and more particularly to a dense video description method based on progressive modal decoupling. Background Technology

[0002] With the explosive growth of visual content such as short videos, surveillance videos, and streaming media, dense video description generation technology has become a core technology in the fields of intelligent content annotation, video retrieval, and intelligent narration. Dense video description requires segmenting multiple video events from long and unedited videos and generating text descriptions for each event. The text descriptions need to fully cover temporal semantic information such as action changes, object interactions, or scene transitions.

[0003] Currently, there are many existing methods for generating dense video descriptions, such as proposal and description pipeline methods, end-to-end sequence generation methods, and retrieval enhancement methods. These three approaches generally assume that more or richer information sources are added to transform them into better video descriptions. However, given a fixed pool of visual and textual information channels, the model does not learn this information in a truly balanced way during training, which can easily lead to over-reliance on visual information during training, thereby reducing the quality of the final generated video description.

[0004] Therefore, how to make the model make balanced use of various information during the training process and improve the accuracy of the generated video descriptions has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a dense video description method based on progressive modal decoupling, which enables the model to make balanced use of multiple types of information during training, thereby improving the accuracy of the generated video description.

[0006] A first aspect of the present invention provides a method for generating dense video descriptions, comprising:

[0007] Obtain initial multimodal data for model training from the training videos;

[0008] Based on the temporal boundary data of the training video, the visual feature data of the initial multimodal data is subjected to temporal localization weighting processing to obtain multimodal data with enhanced geometric boundary perception.

[0009] A modality-aware factor is designed, and semantic boundary decoupling processing is performed on the training video based on the modality-aware factor and semantic transformation cues to obtain the final multimodal feature data;

[0010] The final multimodal feature data is processed using a cross-attention autoregressive mechanism to generate event description data. The event description data is then sorted according to time order to generate an event description sequence.

[0011] Optionally, in one possible implementation of the first aspect, obtaining the initial multimodal data for model training based on the training video includes:

[0012] Based on the training video, visual feature data, retrieval feature data, and label data are obtained. The label data is then text-encoded to obtain text feature data.

[0013] The visual feature data, retrieval feature data, and text feature data are concatenated and combined to obtain the initial multimodal data:

[0014]

[0015] in, The initial multimodal data is the combined representation used for the decoder's cross-attention. For sequence-level concatenation, For visual feature data, To retrieve feature data, For text feature data, The number of sampling frames for visual feature data. To retrieve the number of text features in the feature data, The sequence length of the text feature data. For feature dimensions.

[0016] Optionally, in one possible implementation of the first aspect, the step of performing temporal localization weighted processing on the visual feature data of the initial multimodal data based on the temporal boundary data of the training video to obtain geometrically boundary-aware enhanced multimodal data includes:

[0017] The sub-localization data in the time boundary data are located and associated with the video event frames corresponding to the training video to obtain multiple labeled events;

[0018] The labeled events are sorted according to time order to obtain labeled training data;

[0019] Based on the labeled training data, frame-level weights are determined to update the initial multimodal data, resulting in multimodal data with enhanced geometric boundary awareness.

[0020] Optionally, in one possible implementation of the first aspect, the step of updating the initial multimodal data based on the frame-level weights determined from the labeled training data to obtain geometrically boundary-aware enhanced multimodal data includes:

[0021] The frame-level weights corresponding to each labeled event are determined based on the labeled event and the video duration of the labeled training data:

[0022] in, To label the event, i represents the index of the feature in the current visual frame. Let i be the normalized position of the i-th visual frame on the entire video timeline. Let n be the start time of the nth labeled event. Let N be the end time of the nth labeled event, and N be the total number of labeled events. Let be an interval response function with a sharpness of 10, and D be the duration of the video. To maximize the output of the response function;

[0023] The visual feature data is weighted based on the frame-level weights to obtain weighted visual feature data.

[0024] The initial multimodal data is updated based on the weighted visual feature data to obtain multimodal data with enhanced geometric boundary perception.

[0025] Optionally, in one possible implementation of the first aspect, the step of performing semantic boundary decoupling processing on the training video based on the modality-aware factor and semantic transformation cues to obtain the final multimodal feature data includes:

[0026] Visual modality attenuation processing is performed on the visual feature data corresponding to the training video based on the modality perception factor to obtain visual attenuation feature data;

[0027] Semantic transformation cues are constructed to perform semantic transformation labeling on the labeled training data to obtain enhanced text data;

[0028] The geometric boundary perception enhancement multimodal data is updated based on the visual attenuation feature data and the enhanced text data to obtain the final multimodal feature data.

[0029] Optionally, in one possible implementation of the first aspect, the step of performing visual modal attenuation processing on the visual feature data corresponding to the training video based on the modal awareness factor to obtain visual attenuation feature data includes:

[0030] Based on the first preset value selected during the training phase as the modality perception factor, element-wise multiplication is performed on each visual sub-data in the visual feature data according to the modality perception factor to obtain the attenuation sub-data:

[0031]

[0032] in, For attenuated sub-data, For modal sensing factor, For visual sub-data;

[0033] Visual attenuation feature data are obtained by statistically analyzing the attenuation sub-data.

[0034] Optionally, in one possible implementation of the first aspect, the construction of semantic transformation cues to perform semantic transformation labeling on the labeled training data to obtain enhanced text data includes:

[0035] Obtain the description text corresponding to each labeled event in the labeled training data, perform syntactic analysis on the description text, and obtain the action object pair corresponding to the description text;

[0036] Based on a preset comparison library, semantic mapping is performed on the action object pairs to obtain semantic category data;

[0037] Semantic transformation identification is performed on adjacent semantic category data based on the order of events to determine the boundary strength of labeled events;

[0038] Based on the boundary strength, a semantic transformation clue is constructed, and the semantic transformation clue is transformed to obtain a boundary marker set. The boundary marker set is then concatenated with the text feature data to obtain enhanced text data.

[0039] Optionally, in one possible implementation of the first aspect, the step of performing semantic transition recognition on adjacent semantic category data according to the event arrangement order to determine the boundary strength of the labeled events includes:

[0040] The semantic category data of adjacent action object pairs are compared. When the action category data in the adjacent semantic category data are different and not empty, the first action value is obtained.

[0041] When the object category data in adjacent semantic category data are different and not empty, the value of the first object is obtained;

[0042] When it is determined that the action category data in adjacent semantic category data are the same or there is an empty set, the second action value is obtained;

[0043] When the object category data in adjacent semantic category data are the same or there is an empty set, the second object value is obtained;

[0044] The boundary strength is determined based on the first action value, the second action value, the first object value, and the second object value.

[0045] Optionally, in one possible implementation of the first aspect, determining the boundary strength based on the first action value, the second action value, the first object value, and the second object value includes:

[0046] Based on the first action value and the first object value, the boundary strength is obtained as strong boundary strength;

[0047] The object movement type is obtained based on the second action value and the first object value, and the action conversion type is obtained based on the first action value and the second object value.

[0048] The corresponding boundary strength is determined as the middle boundary strength based on the object movement type and the action transformation type.

[0049] Based on the second action value and the second object value, the boundary strength is determined to be a weak boundary strength.

[0050] Optionally, in one possible implementation of the first aspect, the step of constructing semantic transformation cues based on the boundary strength and transforming the semantic transformation cues to obtain a boundary tag set includes:

[0051] When the boundary strength is determined to be a non-weak boundary strength, a semantic transformation clue is generated based on the boundary strength and the corresponding adjacent action object pair, and the semantic transformation clue is transformed to obtain a boundary transformation mark;

[0052] The boundary transition markers are sorted according to time order to obtain a boundary marker set.

[0053] A second aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program, wherein the computer program is stored in the memory, and the processor executes the computer program to perform the methods described in the first aspect of the present invention and various possible methods related to the first aspect.

[0054] A third aspect of the present invention provides a storage medium storing a computer program, which, when executed by a processor, is used to implement the first aspect of the present invention and various methods possibly involved in the first aspect.

[0055] The beneficial effects of this invention are as follows:

[0056] 1. This invention enables the model to utilize multiple types of information in a balanced manner during training, thereby improving the accuracy of generated video descriptions. Specifically, this invention employs three different training stages to allow the model to achieve standardized fusion of multimodal features and accurate localization of event boundaries, thus improving the accuracy of generated video descriptions. The first stage performs standard training to equip the model with multimodal data understanding capabilities. The second stage uses temporal localization processing to obtain geometric boundary localization information of events in the video, enabling the third stage to decouple semantic boundaries from the video. This reduces the model's over-reliance on visual feature data during training while simultaneously increasing semantic boundary information from the text channel, thereby improving the accuracy of the model's generated video descriptions.

[0057] 2. This invention can achieve semantic boundary decoupling through visual feature attenuation and semantic transformation markers, thereby improving the accuracy of generated video descriptions. Specifically, this invention introduces a modality-aware factor during the training phase to attenuate visual feature data, reducing the model's dependence on visual features. Simultaneously, it determines the boundary strength of video events, improving the accuracy of video event segmentation. Then, semantic transformation markers are generated based on the boundary strength. These markers are combined with textual feature data to obtain enhanced semantic data, improving the model's accuracy in locating video event boundaries and ultimately enhancing the accuracy of the generated video descriptions.

[0058] 3. This invention can ensure the coherence and integrity of the description sequence through cross-attention autoregressive generation and temporal sorting, thereby improving the quality of dense video descriptions. Specifically, this invention employs a cross-attention autoregressive generation method to deeply fuse information from three modalities, thereby determining the core actions, objects, and scene details of each labeled event. It then generates individual event description data that conforms to the event semantics, and strictly follows the natural chronological order of the video to sort the generated individual event description data, forming an event description sequence covering all events in the entire long-term video, thus achieving temporal coherence and logical integrity of the description sequence. Attached Figure Description

[0059] Figure 1 A flowchart illustrating a dense video description method based on progressive modal decoupling provided by this invention;

[0060] Figure 2 A flowchart for acquiring enhanced text data provided by the present invention;

[0061] Figure 3 A schematic diagram illustrating the determination of action object pairs provided by the present invention;

[0062] Figure 4 A flowchart for determining boundary strength provided by the present invention;

[0063] Figure 5 This is a schematic diagram illustrating the principle of progressive modal decoupling provided by the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0066] like Figure 1 As shown, this invention provides a dense video description method based on progressive modal decoupling, comprising:

[0067] S1: Obtain initial multimodal data for model training based on the training video.

[0068] It should be noted that the model training is mainly divided into three stages. In the first stage, the basic model is used with standard configuration, such as 20 training cycles and a learning rate of 3e4, to train the model so that it can establish basic language generation and multimodal understanding capabilities. Therefore, in the first training stage, the initial multimodal data under the standard configuration can be determined based on the training video, so that the initial multimodal data can be trained and adjusted in subsequent training stages to obtain the final training objective and improve the quality of the video descriptions generated by the model.

[0069] It is understandable that the training video is the long-term original video used for model training, and the initial multimodal data is the multimodal data corresponding to the initial training stage, that is, the basic fusion data that integrates multiple features of different dimensions for training the dense video description model.

[0070] In some embodiments, a specific implementation of step S1 (obtaining the initial multimodal data for model training based on the training video) includes:

[0071] S11, based on the training video, visual feature data, retrieval feature data and label data are obtained, and the label data is text-encoded to obtain text feature data.

[0072] Understandably, existing basic models, such as retrieval enhancement models, are used to acquire features from training videos in three dimensions: video content, associated retrieval database, and speech recognition. This yields visual feature data, retrieval feature data, and labeled speech recognition data, which are then input into a text encoder to obtain the corresponding text feature data.

[0073] For example, visual frame features can be extracted by comparing them with large pre-trained models of language images. ,in This refers to the visual feature data, specifically the global visual feature matrix of the entire video. The visual feature data corresponding to the i-th sampled frame is the single-frame feature vector. Let T be the number of sampled frames and d be the feature dimension. ,in, To retrieve feature data, namely the text feature matrix retrieved from hierarchical memory, Let j be the feature vector of the j-th text unit. The quantity of text feature data retrieved is the quantity of text features obtained from external memory. The labeled data is a tokenized ASR input of length L, where L is the total length of the tokenized ASR text sequence. This represents the Lth text tag. After obtaining the tag data, it is processed by a T5 encoder. Processing the text stream of labeled data, i.e. ,in, Text feature data The T5 encoder function can transform discrete text tags into continuous high-dimensional feature vectors.

[0074] Among them, visual feature data consists of image and motion feature data extracted from video frames, retrieval feature data consists of associated semantic feature data matched with an external retrieval library, labeled data consists of data obtained by speech recognition of training videos, and text feature data consists of feature data after text encoding of labeled data.

[0075] S12, the visual feature data, retrieval feature data, and text feature data are concatenated and combined to obtain initial multimodal data:

[0076]

[0077] in, The initial multimodal data is the combined representation used for the decoder's cross-attention. For sequence-level concatenation, For visual feature data, To retrieve feature data, For text feature data, The number of sampling frames for visual feature data. To retrieve the number of text features in the feature data, The sequence length of the text feature data. For feature dimensions.

[0078] It is understandable that combining the three types of heterogeneous feature data—visual, retrieval, and text—results in complete initial multimodal data, enabling the model to understand and learn multimodal data in the first training phase and improving the quality of dense video descriptions.

[0079] S2, based on the temporal boundary data of the training video, perform temporal localization weighting processing on the visual feature data of the initial multimodal data to obtain geometrically boundary-aware enhanced multimodal data.

[0080] It should be noted that the initial multimodal data obtained in the first training phase extracts visual features from the entire training video without distinguishing the temporal boundaries of different events within the long video. This can easily lead to the model's inability to accurately identify the segmentation nodes of different video events, resulting in descriptions that are prone to event confusion and temporal discrepancies. Therefore, in the second training phase, the temporal boundary data of the training video can be used to temporally constrain the visual features in the initial multimodal data, clarify the range of visual features corresponding to different video events, and then the initial multimodal data can be updated after targeted optimization of the visual features to obtain multimodal data with enhanced geometric boundary awareness, thereby improving the model training effect and the quality of event description generation.

[0081] It is understandable that temporal boundary data refers to temporal information data that marks the start and end times of different events within the training video, while geometric boundary perception-enhanced multimodal data refers to multimodal data that has been optimized for model training after visual feature temporal localization.

[0082] In some embodiments, the specific implementation of step S2 (the temporal localization weighting processing of the visual feature data of the initial multimodal data based on the temporal boundary data of the training video to obtain geometrically boundary-aware enhanced multimodal data) includes:

[0083] S21 locates and associates each sub-localization data in the time boundary data with the video event frames corresponding to the training video to obtain multiple labeled events.

[0084] Understandably, the temporal boundary data contains multiple sub-localization data, each corresponding to the start and end times of a video event. By associating these sub-localization data with specific video event frames of the training video, the corresponding video content within each time interval is clearly defined, thereby dividing the video into multiple independent labeled events. This enables event segmentation of long-term training videos, allowing the model to clearly identify the visual range of different events, avoid event confusion, and improve the accuracy of subsequent video descriptions.

[0085] Among them, the sub-location data are the start and end time data corresponding to a single video event, which are split from the time boundary data; the video event frame is a series of video frames corresponding to a certain event in the training video; and the labeled event is an independent video event associated through time localization.

[0086] S22, Sort the labeled events according to time order to obtain labeled training data.

[0087] Understandably, since the multiple labeled events are scattered, and dense video descriptions need to generate event descriptions according to the time sequence of video playback, all labeled events can be arranged in an orderly manner according to the natural time sequence of the training video to form labeled training data. This makes it easier for the temporal sequence of labeled events to be consistent with the actual playback logic of the video, allowing the model to learn the temporal relationship between events.

[0088] Wherein, the time sequence is the natural playback order of the training video, and the labeled training data is the set of all labeled events arranged in chronological order.

[0089] S23, based on the labeled training data, determine the frame-level weights to update the initial multimodal data, thereby obtaining geometrically boundary-aware enhanced multimodal data.

[0090] In some embodiments, a specific implementation of step S23 (updating the initial multimodal data based on the frame-level weights determined from the labeled training data to obtain geometrically boundary-aware enhanced multimodal data) includes:

[0091] S231, determine the frame-level weights corresponding to each labeled event based on the labeled event and the video duration of the labeled training data:

[0092] in, To label the event, i represents the index of the feature in the current visual frame. Let i be the normalized position of the i-th visual frame on the entire video timeline. Let n be the start time of the nth labeled event. Let N be the end time of the nth labeled event, and N be the total number of labeled events. Let be an interval response function with sharpness = 10, and D be the duration of the training video. This is to maximize the output of the response function.

[0093] Understandably, since a video contains a large number of frames but few keyframes that truly describe the temporal boundaries, if all frames are treated with the same learning weights, the model will be overwhelmed by a large amount of unambiguous background information, leading to inaccurate localization. Therefore, weighted processing can be used to make the model more biased towards keyframes when calculating the loss function. Calculating frame-level weights is to distinguish the important relationships between video frames. Following the privileged temporal supervision paradigm, the suppressed annotations are used as privileged information. During the training phase, the knowledge of what is important is forcibly etched into the model's parameters, so that the model can still accurately locate itself based on these parameters even when visual obstruction occurs.

[0094] Among them, the frame-level weight is the weight value of the visual feature importance corresponding to each labeled event, and the video duration is the length of the entire training video corresponding to the labeled training data.

[0095] S232, The visual feature data is weighted based on the frame-level weights to obtain weighted visual feature data.

[0096] Understandably, the visual feature data in the initial multimodal data did not distinguish the importance of different events. By weighting the frame-level weights with the visual feature data of the corresponding labeled events, the visual features of important labeled events are strengthened while the visual features of secondary events are weakened, making the visual feature data more targeted. This avoids the model from overlearning irrelevant visual information, while highlighting the visual feature data of key events and improving the model's ability to identify key events.

[0097] The weighted calculation involves multiplying and summing the frame-level weights with the corresponding visual feature data; that is, the visual features are reweighted. , The weighted visual feature data is the visual feature data after frame-level weight optimization.

[0098] S233, the initial multimodal data is updated based on the weighted visual feature data to obtain multimodal data with enhanced geometric boundary perception.

[0099] Understandably, by replacing the original visual feature data in the initial multimodal data with the optimized weighted visual feature data, the initial multimodal data is updated to obtain geometrically boundary-aware enhanced multimodal data. The updated geometrically boundary-aware enhanced multimodal data retains the advantages of retrieval features and text features, and solves the problem of visual feature temporal ambiguity through temporal localization and weighted optimization. Therefore, the second training stage increases the model's ability to perceive the boundaries of visual sets and improves the generation quality of event descriptions.

[0100] S3. Design a modality-aware factor, and perform semantic boundary decoupling processing on the training video based on the modality-aware factor and semantic transformation cues to obtain the final multimodal feature data.

[0101] It should be noted that while the first two training phases enable the current model to understand multimodal data and resolve the issue of unclear semantic boundaries across different time periods, there is still an over-reliance on visual feature data during model training, which affects the accuracy of dense video descriptions. Therefore, after completing geometric boundary localization, a third training phase can be initiated to decouple semantic boundaries. This reduces the learning of visual feature data during training and injects semantic transfer training through the text channel, thereby reducing the dependence on visual feature data and allowing the released text channel to learn corresponding semantic prompts, thus improving the accuracy of generating dense video descriptions.

[0102] Among them, the modality perception factor is a preset value for adjusting the attenuation of visual features to balance the weights of multimodal information. The final multimodal feature data is the multimodal training data of the model after semantic decoupling optimization.

[0103] In some embodiments, the specific implementation of step S3 (the semantic boundary decoupling processing of the training video based on the modality-aware factor and the semantic transformation cue training video to obtain the final multimodal feature data) includes:

[0104] S31, Visual modal attenuation processing is performed on the visual feature data corresponding to the training video based on the modal perception factor and semantic conversion cues to obtain visual attenuation feature data.

[0105] Understandably, since the first two models rely too heavily on visual information during training, the learning of retrieval features and text features is easily neglected, which in turn affects the accuracy of the generated event descriptions. Therefore, the visual feature data of the model training can be subjected to visual modality attenuation processing through modality perception factor, which can appropriately reduce the representation intensity of visual features, weaken the model's over-reliance on visual information, and achieve a weight balance of multimodal information.

[0106] Among them, visual modality attenuation processing is a processing operation that reduces the intensity of visual feature representation through modality perception factor, and visual attenuation feature data is visual feature data after modality perception factor attenuation processing.

[0107] In some embodiments, the specific implementation of step S31 (performing visual modal attenuation processing on the visual feature data corresponding to the training video according to the modal perception factor to obtain visual attenuation feature data) includes:

[0108] S311, based on the first preset value selected during the training phase as the modality perception factor, and performing element-wise multiplication on each visual sub-data in the visual feature data according to the modality perception factor, to obtain the attenuation sub-data:

[0109]

[0110] in, For attenuated sub-data, For modal sensing factor, This is visual sub-data.

[0111] Understandably, in order to reduce the model's over-reliance on visual feature data during the training phase, modality acuity factors are used only during the training phase to attenuate the visual feature data. It is 0, but during the formal model inference phase, the modality-aware factor... The data is restored to 1. During deployment, complete, undiminished visual feature data is used. Therefore, during the training phase, a first preset value is selected as the modality perception factor to attenuate the visual feature data, resulting in attenuated sub-data, which is then used for visual scaling training of the model.

[0112] The first preset value is a pre-set modality perception factor value adapted to the current training stage, such as the modality perception factor. Visual sub-data refers to a single visual feature data corresponding to the visual feature data, and attenuation sub-data refers to the feature data obtained after attenuation of a single visual sub-data by the modal perception factor.

[0113] S312, statistical analysis of the attenuation sub-data yields visual attenuation feature data.

[0114] Understandably, all attenuation sub-data are aggregated and statistically analyzed to obtain visual attenuation feature data in order to complete the third stage of model training.

[0115] S32, construct semantic transformation cues to perform semantic transformation labeling on the labeled training data to obtain enhanced text data.

[0116] It should be noted that Visual Modal Attenuation (VMA) alleviates the dependence on visual features, but does not provide any new structural information to the text channel. Therefore, structured boundary semantics can be conveyed through the existing text channel based on a parameter-free, offline extractable semantic transformation tag signal without changing the model or architecture to enhance the model's understanding of events, thereby improving the accuracy of the generated video event description.

[0117] Understandably, the labeled training data only contains geometric temporal location information related to events, but lacks clear semantic differences and boundary features between events. This makes it difficult for the model to accurately distinguish the semantic boundaries of different events, leading to semantic ambiguity and disjointed event descriptions. Therefore, as... Figure 2As shown in the flowchart, to obtain enhanced text data, semantic transformation and labeling can be performed on the descriptive text of each labeled event in the labeled training data, thereby mining the semantic features of the event to generate enhanced text data containing boundary semantics. This is to supplement the text channel with structured information, thereby balancing the weight of multimodal information in conjunction with visual attenuation processing, allowing the model to more clearly identify the semantic boundaries of the event and improve the accuracy of description generation.

[0118] Among them, enhanced text data refers to optimized text feature data, that is, text feature data containing structured information such as semantic categories and boundary markers.

[0119] In some embodiments, the specific implementation of step S32 (the construction of semantic transformation cues to perform semantic transformation labeling on the labeled training data to obtain enhanced text data) includes:

[0120] S321, Obtain the description text corresponding to each labeled event in the labeled training data, perform syntactic analysis on the description text, and obtain the action object pair corresponding to the description text.

[0121] It is understandable that, since the descriptive text corresponding to each labeled event contains the core semantic information of the event, such as the action and the object of the action, we can use existing syntactic analysis tools to decompose the descriptive text corresponding to each labeled event, thereby separating the core elements of the action and the object of the action in the text, and obtaining the action object pair corresponding to each labeled event. This makes it easier for the model to accurately capture the core semantics of the event.

[0122] For example, dependency parsing can be used to extract an action-object pair from each description:

[0123]

[0124] in, For action, For objects, To describe the text, For custom syntax parsing functions, To return to its basic form, To return the root verb in the dependency tree, To return its direct object and compound modifiers.

[0125] It's easy to understand that when the parser fails to identify the root verb or the root verb has no direct object, settings can be configured... The heading will be treated as unparsed in all subsequent steps. For headings containing multiple root clauses, only the first root verb will be used.

[0126] Among them, the description text is the speech-to-text or manually annotated text corresponding to the labeled event, the syntactic analysis is the processing method of deconstructing the grammatical structure of the description text and extracting the core semantic elements, and the action-object pair is a semantic combination containing an action and an object, such as cutting onions, where the action is cutting and the object is onions.

[0127] S322, based on a preset comparison library, perform semantic mapping on the action object pair to obtain semantic category data.

[0128] Understandably, the preset comparison library is a manually constructed dictionary containing action groups and object categories. In order to reduce the interference of redundant data in action-object pairs, the extracted action-object pairs can be semantically mapped based on the preset comparison library to obtain the corresponding semantic category data, which facilitates the subsequent determination of the boundary strength of each labeled event, thereby improving the accuracy of the event description generated by the model.

[0129] The preset comparison library is a pre-configured database containing mapping relationships between actions, objects and corresponding standard semantic categories. Semantic mapping is the processing operation that matches the original action-object pairs to the preset standard semantic categories. The semantic category data is the standardized semantic category information obtained after the action-object pairs are mapped.

[0130] For example, such as Figure 3 The diagram illustrates how action object pairs are identified based on the descriptive text of labeled events and then semantically mapped and grouped. For the YouCook2 instructional video dataset, 10 action groups can be defined as cutting, adding, mixing, cooking, serving, removing, spreading, shaping, covering, and flipping, along with 9 object categories: vegetables, meat, seafood, dairy products, grains, sauces, seasonings, mixtures, and tools. These categories serve as a pre-defined comparison library to map the corresponding action object pairs, resulting in semantic category data, i.e., each extracted pair. All are mapped to semantic categories through manually constructed dictionaries:

[0131]

[0132] in, As an action semantic category, As an action semantic category, Mapping the action lemma to One of the action groups, Map the cleaned-up objects to One of the object categories, clean() removes metrics and modifiers to reduce noise.

[0133] S323, based on the order of events, perform semantic transformation recognition on adjacent semantic category data to determine the boundary strength of labeled events.

[0134] Understandably, since the greater the semantic difference, the clearer the event boundary, and the smaller the semantic difference, the more blurred the event boundary, the semantic category data of adjacent labeled events can be compared one by one according to the chronological order of the labeled events. This allows the identification of the semantic transition between adjacent events, and the boundary strength corresponding to each labeled event can be determined based on the degree of semantic transition. This enables the generation of enhanced text features for boundary markers, allowing the model to accurately distinguish the semantic boundaries of different events, reducing the problem of confusion and overlap in event descriptions, and thus improving the accuracy of the generated video event descriptions.

[0135] Among them, the event arrangement order is the chronological order of each labeled event in the labeled training data, the semantic transformation recognition is the processing operation of comparing the semantic category data of adjacent events and judging the degree of semantic difference between them, and the boundary strength is the clarity of the semantic boundary between adjacent labeled events, such as strong, medium and weak boundary strength.

[0136] It is not difficult to understand that, as Figure 4 As shown, two adjacent semantic category data A and semantic category data B are compared to determine the action value and object value. The action value includes the first action value and the second action value, and the object value includes the first object value and the second object value. The corresponding boundary strength is obtained according to different combinations of values.

[0137] It's worth noting that not all event transitions carry the same semantic weight. When both the action and the object change—for example, from "chopping onions" to "stirring sauce"—this transition has a stronger boundary than simply applying the same action to different ingredients. To reflect this distinction, each boundary can be divided into multiple intensity levels based on the changes in action groups and object categories between adjacent events. For each pair of adjacent events (n, n+1), two binary indicators can be defined to measure whether a transition has occurred in each semantic dimension:

[0138]

[0139]

[0140] in, For changes in action, The requirement is that there are identified action groups on both sides, and these groups are different. For object changes, It requires that there are identified object groups on both sides, and these groups are different; otherwise, the value is 0.

[0141] Then the boundary strength is divided into:

[0142]

[0143] in, The boundary strength of adjacent events is determined by the following steps for specific analysis.

[0144] In some embodiments, the specific implementation of step S323 (the semantic transformation recognition of adjacent semantic category data according to the event arrangement order, and the determination of the boundary strength of the labeled events) includes:

[0145] S3231, compare the semantic category data of adjacent action object pairs, and when it is determined that the action category data in the adjacent semantic category data are different and not empty, obtain the first action value.

[0146] Understandably, the first action value is a judgment value that marks the difference between adjacent event action category data and is not empty. It can be preset, such as 1.

[0147] S3232, when it is determined that the object category data in adjacent semantic category data are different and not empty, the value of the first object is obtained.

[0148] Understandably, the first object value is a value used to determine whether adjacent event object category data is different and not empty. It can be preset, for example, to be 1.

[0149] S3233, when it is determined that the action category data in adjacent semantic category data are the same or there is an empty set, the second action value is obtained.

[0150] Understandably, the second action value is a judgment value that marks adjacent event action category data as being the same or empty, and it can be preset, such as 0.

[0151] It is easy to understand that when extracting action object pairs, if the model cannot clearly determine the corresponding semantic information or accurately map the category data for the corresponding event, then the corresponding action object pair and object category data can be determined to be empty.

[0152] S3234, when it is determined that the object category data in adjacent semantic category data are the same or there is an empty set, the second object value is obtained.

[0153] Understandably, the value of the second object is a value used to determine whether adjacent event objects have the same category data or an empty set. This value can be preset, for example, to 0.

[0154] S3235, determine the boundary strength based on the first action value, the second action value, the first object value, and the second object value.

[0155] It is understandable that the first action value and the second action value reflect the semantic differences in the action dimension of adjacent events, and the first object value and the second object value reflect the semantic differences in the object dimension of adjacent events. Therefore, the two can be combined to comprehensively determine the degree of semantic transformation of adjacent events. That is, by integrating these four judgment values ​​according to the preset judgment rules such as numerical combination matching, the clarity of the semantic boundary of adjacent labeled events can be clarified, and the corresponding boundary strength can be determined, so that the subsequent model can accurately identify the semantic boundary of the event.

[0156] In some embodiments, a specific implementation of step S3235 (determining the boundary strength based on the first action value, the second action value, the first object value, and the second object value) includes:

[0157] S32351, Based on the first action value and the first object value, the boundary strength is obtained as strong boundary strength.

[0158] It is understandable that the first action value corresponds to different and non-empty action category data of adjacent events, and the first object value corresponds to different and non-empty object category data of adjacent events. When both conditions are met, it means that the core action and core object of adjacent labeled events have changed significantly. This indicates that the degree of semantic transformation is the greatest, that is, the boundary between events is the clearest. Therefore, the boundary strength at this time can be determined to be strong boundary strength, and this position is clearly defined as the clear dividing point between two independent events, so as to generate the corresponding boundary markers in the future.

[0159] Among them, strong boundary strength is the boundary strength level where the semantic boundary between adjacent labeled events is clearest and the semantic change is most obvious.

[0160] S32352, Obtain the object movement type based on the second action value and the first object value, and obtain the action conversion type based on the first action value and the second object value.

[0161] It is understandable that the second action value corresponds to the same or empty action category data of the adjacent event. Combined with the first object value, which is different and not empty, it indicates that the core action of the adjacent event has not changed, but the core object has changed. This semantic change corresponds to the event scenario of object transformation. Therefore, the semantic type of this situation can be determined as object movement type. The first action value corresponds to the judgment condition that the action category data is different and not empty. Combined with the judgment condition that the second object value corresponds to the judgment condition that the object category data is the same or empty, it indicates that the core object of the adjacent event has not changed, but the core action has changed. This semantic change corresponds to the event scenario of action conversion. Therefore, the semantic type of this situation can be determined as action conversion type. Both types belong to the case of moderate semantic change.

[0162] Among them, the object movement type is a semantic type in which the action in adjacent events does not change but the object changes, and the action transformation type is a semantic type in which the object in adjacent events does not change but the action changes.

[0163] S32353, the corresponding boundary strength is obtained as the middle boundary strength based on the object movement type and the action transformation type.

[0164] It is understandable that both object movement type and action transformation type belong to adjacent events where only one dimension, namely the action or object, undergoes a semantic change. The degree of semantic change is between the change of both action and object and the change of both action and object. The clarity of the event boundary is moderate. Therefore, the boundary strength corresponding to these two semantic transformation types can be determined as medium boundary strength, thus distinguishing them from strong boundary strength, so that the model can identify event boundaries of different degrees.

[0165] Among them, the middle boundary strength is the boundary strength level with moderate semantic clarity and moderate semantic transformation between adjacent labeled events.

[0166] S32354, based on the second action value and the second object value, the boundary strength is obtained as a weak boundary strength.

[0167] It is understandable that the second action value corresponds to the same or empty action category data of adjacent events, and the second object value corresponds to the same or empty object category data of adjacent events. When both conditions are met, it means that the core action and core object of adjacent labeled events have not changed significantly, the degree of semantic change is minimal, the boundary between events is most blurred, and they may even belong to different segments of the same event. Therefore, it can be determined that the boundary strength at this time is weak boundary strength, so that the model can clearly understand that there is no need to perform obvious event segmentation at this position during training, and avoid the model from over-segmenting events and causing redundancy in description.

[0168] Among them, the weak boundary strength is the boundary strength level where the semantic boundary between adjacent labeled events is the most ambiguous and the semantic change is the least obvious.

[0169] S324, construct semantic transformation cues based on the boundary strength, transform the semantic transformation cues to obtain a boundary marker set, and concatenate the boundary marker set with the text feature data to obtain enhanced text data.

[0170] It is understandable that step S323 has determined the boundary strength of each adjacent labeled event. The semantic transformation of events corresponding to weak boundary strengths is not obvious, thus eliminating the need to generate boundary markers. Therefore, boundary transformation markers are only generated for non-weak boundary strengths, i.e., strong and medium boundary strengths. These markers are then sequentially integrated into a boundary marker set, which is then concatenated with the previously obtained text feature data to supplement the structured boundary semantic information of the text channels. This approach allows the text features to carry event boundary information without altering the model architecture, facilitating the model's learning and training of multimodal information in conjunction with visual attenuation feature data, enhancing the model's ability to recognize event boundaries, and improving the accuracy of the generated video descriptions. The boundary marker set is a set containing boundary transformation markers corresponding to non-weak boundary strengths, arranged chronologically. Concatenation is a processing operation that combines the boundary marker set with the text feature data along the feature dimension. ,

[0171]

[0172] in, To enhance text data, For the k-th element, tok() is the T5 token. The Vid2Seq time slice marker, also known as the boundary transition marker, is anchored to the boundary timestamps (i.e., the end time of event N and the start time of event N+1). Let be the end time of event N, and let be the start time of event N+1. This is the Mth boundary transformation mark in the boundary mark set, where M is the sequence number of the boundary transformation marks in the boundary mark set.

[0173] In some embodiments, the specific implementation of step S324 (constructing semantic transformation clues based on the boundary strength and transforming the semantic transformation clues to obtain a boundary marker set) includes:

[0174] S3241, when the boundary strength is determined to be a non-weak boundary strength, a semantic transformation clue is generated based on the boundary strength and the corresponding adjacent action object pair, and the semantic transformation clue is transformed to obtain a boundary transformation mark.

[0175] It should be noted that since there is a significant semantic shift between adjacent events corresponding to non-weak boundary strength, corresponding boundary transition markers need to be generated to represent this semantic shift. However, since the semantic shift between weak boundary strength is not obvious, there is no need to generate boundary transition markers to avoid redundant markers affecting model training. When generating semantic transition cues, the boundary strength level and the action object pair of the corresponding adjacent labeled events can be combined so that the boundary transition markers contain both the clarity of the boundary and the association with the core semantics of the event. This allows the model to accurately identify the event boundary and the type of transition through the boundary transition markers, thus providing specific boundary semantic information for text feature enhancement.

[0176] Understandably, the boundary phrase b(n,n+1) represented as a plain text string is constructed as follows:

[0177]

[0178] Where n is the nth labeled event in the training data, and n+1 is the (n+1)th labeled event in the training data. For prefix functions, As a description function, the boundary strength in the prefix function refers to the boundary strength corresponding to adjacent labeled events. This allows the generation of prefix information corresponding to the boundary strength, and the generation of boundary descriptions for adjacent labeled events through the description function. This constructs boundary strength type prefixes and title-style descriptions as semantic transformation cues. Furthermore, the semantic transformation cues corresponding to the non-weak boundary strengths in the video are sorted by time, and each boundary semantic transformation cues is wrapped with a Vid2Seq time warehouse tag anchored to the boundary timestamp, thus obtaining the boundary transformation tag. Among them, semantic transformation cues are boundary type hints formed after the action category and object category change between adjacent events. Non-weak boundary strength is a general term for strong boundary strength and medium boundary strength. Boundary transformation markers are standardized markers of the semantic boundary and transformation type of adjacent events. For example, when the boundary strength is strong boundary strength, the corresponding semantic transformation cues can be: strong boundary strength ∆A=1, ∆O=1: boundary, after cutting the onion, stir the sauce. When it is an action transformation type in the medium boundary strength, the corresponding semantic transformation cues can be: medium boundary strength action type ∆A=1, ∆O=0: action transformation: go around and stir the onion. When it is an object movement type, the corresponding semantic transformation cues can be: medium boundary strength object type ∆A=0, ∆O=1: object movement: continue cutting the garlic.

[0179] S3242, Sort the boundary transformation markers according to time order to obtain a boundary marker set.

[0180] It is understandable that boundary transformation markers correspond to the semantic shifts of adjacent labeled events, and since labeled events are arranged in chronological order, all generated boundary transformation markers can be sorted according to the natural chronological order of labeled events so that the temporal sequence of the boundary markers is consistent with the temporal sequence of video events. The sorted boundary transformation markers are then aggregated and integrated to obtain a structured boundary marker set, which facilitates subsequent concatenation with text feature data. That is, the boundary transformation markers in the boundary marker set are appended to the original ASR marker sequence in chronological order to ensure the temporal sequence of the enhanced text data is reasonable, allowing the model to understand event boundary changes in chronological order. S33, the multimodal data for geometric boundary perception enhancement is updated based on the visual attenuation feature data and enhanced text data to obtain the final multimodal feature data.

[0181] Understandably, the geometric boundary perception-enhanced multimodal data includes visual feature data, retrieval feature data, and text feature data. After the above processing steps, the visual feature data has been optimized into visual attenuation feature data, and the text feature data has been optimized into enhanced text data. Therefore, the original visual and text features in the geometric boundary perception-enhanced multimodal data can be replaced with the optimized visual attenuation feature data and enhanced text data to complete the data update and obtain the final multimodal feature data. In the process of obtaining the final multimodal feature data, the weights of multimodal information are balanced, which not only solves the problem of over-reliance on vision but also supplements the semantics of event boundaries, so that the model can accurately identify video events and generate high-quality video descriptions.

[0182] It is not difficult to understand that, as Figure 5 The diagram illustrates the principle of progressive modal decoupling. In this process, visual feature data is geometrically weighted by frame-level weighting to enhance the geometric boundary perception capability of multimodal data, resulting in multimodal data with enhanced geometric boundary perception. Furthermore, semantic transformation cues are used to decouple the perception of multimodal data, thereby improving the quality of video description.

[0183] S4, perform cross-attention autoregression on the final multimodal feature data to generate event description data, sort the event description data based on time order, and generate an event description sequence.

[0184] Understandably, since the final multimodal feature data already possesses balanced multimodal information and clear event boundary semantics, a cross-attention mechanism can be used to enable the model to simultaneously focus on visual, retrieval, and textual modal information, thereby deeply fusing multimodal information. Then, an autoregressive approach is used to generate event description data corresponding to each labeled event, i.e., the text description of a single event. Since the labeled events are arranged in chronological order, all generated event description data need to be sorted according to the video's chronological order to form an event description sequence, thereby achieving dense video description generation for long videos and improving the accuracy and completeness of video descriptions.

[0185] Among them, cross-attention autoregression is a generation method that can integrate multimodal information and generate sequence data one by one. It is an existing technology and will not be elaborated here. Event description data is the text description information corresponding to a single labeled event. Event description sequence is a complete set of text descriptions covering all events in the entire training video, arranged in chronological order.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dense video description method based on progressive modal decoupling, characterized in that, include: Obtain initial multimodal data for model training from the training videos; Based on the temporal boundary data of the training video, the visual feature data of the initial multimodal data is subjected to temporal localization weighting processing to obtain multimodal data with enhanced geometric boundary perception. A modality-aware factor is designed, and semantic boundary decoupling processing is performed on the training video based on the modality-aware factor and semantic transformation cues to obtain the final multimodal feature data; The final multimodal feature data is processed using a cross-attention autoregressive mechanism to generate event description data. The event description data is then sorted according to time order to generate an event description sequence.

2. The method according to claim 1, characterized in that, The step of obtaining initial multimodal data for model training based on training videos includes: Based on the training video, visual feature data, retrieval feature data, and label data are obtained. The label data is then text-encoded to obtain text feature data. The visual feature data, retrieval feature data, and text feature data are concatenated and combined to obtain the initial multimodal data: in, The initial multimodal data is the combined representation used for the decoder's cross-attention. For sequence-level concatenation, For visual feature data, To retrieve feature data, For text feature data, The number of sampling frames for visual feature data. To retrieve the number of text features in the feature data, The sequence length of the text feature data. For feature dimensions.

3. The method according to claim 2, characterized in that, The visual feature data of the initial multimodal data is subjected to temporal localization weighting processing based on the temporal boundary data of the training video to obtain geometrically boundary-aware enhanced multimodal data, including: The sub-localization data in the time boundary data are located and associated with the video event frames corresponding to the training video to obtain multiple labeled events; The labeled events are sorted according to time order to obtain labeled training data; Based on the labeled training data, frame-level weights are determined to update the initial multimodal data, resulting in multimodal data with enhanced geometric boundary awareness.

4. The method according to claim 3, characterized in that, The step of determining frame-level weights based on the labeled training data and updating the initial multimodal data to obtain geometrically boundary-aware enhanced multimodal data includes: The frame-level weights corresponding to each labeled event are determined based on the labeled event and the video duration of the labeled training data: in, To label the event, i represents the index of the feature in the current visual frame. Let i be the normalized position of the i-th visual frame on the entire video timeline. Let n be the start time of the nth labeled event. Let N be the end time of the nth labeled event, and N be the total number of labeled events. Let be an interval response function with a sharpness of 10, and D be the duration of the video. To maximize the output of the response function; The visual feature data is weighted based on the frame-level weights to obtain weighted visual feature data. The initial multimodal data is updated based on the weighted visual feature data to obtain multimodal data with enhanced geometric boundary perception.

5. The method according to claim 4, characterized in that, The semantic boundary decoupling process performed on the training video based on the modality-aware factor and semantic transformation cues yields the final multimodal feature data, including: Visual modality attenuation processing is performed on the visual feature data corresponding to the training video based on the modality perception factor to obtain visual attenuation feature data; Semantic transformation cues are constructed to perform semantic transformation labeling on the labeled training data to obtain enhanced text data; The geometric boundary perception enhancement multimodal data is updated based on the visual attenuation feature data and the enhanced text data to obtain the final multimodal feature data.

6. The method according to claim 5, characterized in that, The step of performing visual modal attenuation processing on the training video based on the modality perception factor to obtain visual attenuation feature data includes: Based on the first preset value selected during the training phase as the modality perception factor, element-wise multiplication is performed on each visual sub-data in the visual feature data according to the modality perception factor to obtain the attenuation sub-data: in, For attenuated sub-data, For modal sensing factor, For visual sub-data; Visual attenuation feature data are obtained by statistically analyzing the attenuation sub-data.

7. The method according to claim 5, characterized in that, The construction of semantic transformation cues performs semantic transformation labeling on the labeled training data to obtain enhanced text data, including: Obtain the description text corresponding to each labeled event in the labeled training data, perform syntactic analysis on the description text, and obtain the action object pair corresponding to the description text; Based on a preset comparison library, semantic mapping is performed on the action object pairs to obtain semantic category data; Semantic transformation identification is performed on adjacent semantic category data based on the order of events to determine the boundary strength of labeled events; Based on the boundary strength, a semantic transformation clue is constructed, and the semantic transformation clue is transformed to obtain a boundary marker set. The boundary marker set is then concatenated with the text feature data to obtain enhanced text data.

8. The method according to claim 7, characterized in that, The step of performing semantic transformation recognition on adjacent semantic category data based on the event arrangement order to determine the boundary strength of labeled events includes: The semantic category data of adjacent action object pairs are compared. When the action category data in the adjacent semantic category data are different and not empty, the first action value is obtained. When the object category data in adjacent semantic category data are different and not empty, the value of the first object is obtained; When it is determined that the action category data in adjacent semantic category data are the same or there is an empty set, the second action value is obtained; When the object category data in adjacent semantic category data are the same or there is an empty set, the second object value is obtained; The boundary strength is determined based on the first action value, the second action value, the first object value, and the second object value.

9. The method according to claim 8, characterized in that, The determination of boundary strength based on the first action value, the second action value, the first object value, and the second object value includes: Based on the first action value and the first object value, the boundary strength is obtained as strong boundary strength; The object movement type is obtained based on the second action value and the first object value. The action transition type is then obtained based on the first action value and the second object value. The corresponding boundary strength is determined as the middle boundary strength based on the object movement type and the action transformation type. Based on the second action value and the second object value, the boundary strength is determined to be a weak boundary strength.

10. The method according to claim 9, characterized in that, The step of constructing semantic transformation clues based on the boundary strength and transforming the semantic transformation clues to obtain a boundary marker set includes: When the boundary strength is determined to be a non-weak boundary strength, a semantic transformation clue is generated based on the boundary strength and the corresponding adjacent action object pair, and the semantic transformation clue is transformed to obtain a boundary transformation mark; The boundary transition markers are sorted according to time order to obtain a boundary marker set.

Images