Video description generation method and system based on hypergraph convolutional network, and computer medium

Abstract

The invention provides a video description generation method and system based on a hypergraph convolutional network, and a computer medium. The method comprises the following steps: extracting multi-modal features of video multi-modal data by using a preset large model multi-modal feature extraction network; the extracted multi-modal features are fused, and the fused multi-modal features are obtained; performing hypergraph modeling on the fused multi-modal features by using a preset hypergraph convolutional network to obtain a hypergraph; processing the hypergraph by using a preset time division network, deleting video contents irrelevant to the query text, and performing event division on relevant video contents; and generating a video description statement for each divided event by using a preset video description generation network. According to the method, under the condition that excessive operation is not introduced, the density of semantic information connection between different modes is enhanced in the multi-mode data fusion stage, deep semantic relation in the multi-mode data is obtained, semantic information with finer granularity can be captured, and therefore a high-quality video description text is generated.

Description

Technical Field

[0001] The present invention relates to the field of cross-media generation technology, and more specifically, to a video description generation method, system and computer medium based on a hypergraph convolutional network. Background Art

[0002] Instructional videos provide us with rich visual, auditory, and language information, allowing us to master the specific steps to solve relevant daily problems by watching instructional videos. More and more people rely on them to learn new skills and solve daily problems. For example, people can repair a damaged brick wall or learn how to cook delicious food by watching videos. This phenomenon inspired us to explore whether machines can understand instructional videos and summarize the instructional steps of instructional videos in sequence like humans. Previous work has attempted to use low-level visual features to learn and generate step descriptions, but it is difficult to extract semantic concepts in the picture using visual features alone. With the rapid development of deep learning technology, large models have achieved remarkable results in fields such as natural language processing (NLP) and computer vision (CV). By utilizing the powerful feature extraction capabilities of large models, exploring the deep semantic connections between different modal features, and further extracting deep semantic information in the video, accurate video descriptions can be generated.

[0003] The prior art discloses a video description system based on a graph convolutional network. After sampling the video, the system obtains video features, inputs them into the graph convolutional network to extract semantic information, and finally generates a sentence description that matches the video content. However, when the system performs graph relationship modeling on multimodal data, it cannot well mine the deep semantic connections between multimodal data. Summary of the invention

[0004] One of the purposes of the present invention is to provide a video description generation method based on a hypergraph convolutional network to solve the technical problem that the prior art cannot well mine the deep semantic connections between multimodal data; the second purpose of the present invention is to provide a video description generation system based on a hypergraph convolutional network; the third purpose of the present invention is to provide a computer medium.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A first aspect of the present invention provides a method for generating video description based on a hypergraph convolutional network, comprising the following steps:

[0007] The multimodal features of the video multimodal data are extracted using a preset large-model multimodal feature extraction network;

[0008] The extracted multimodal features are fused to obtain fused multimodal features;

[0009] Using a preset hypergraph convolutional network to perform hypergraph modeling on the fused multimodal features to obtain a hypergraph;

[0010] Processing the hypergraph using a preset time division network, deleting video content irrelevant to the query text and dividing relevant video content into events;

[0011] The preset video description generation network is used to generate a video description sentence for each divided event.

[0012] Among the above technical means, a video description generation method based on a hypergraph convolutional network is provided to address the fact that existing methods cannot efficiently utilize the rich deep semantic connections between multimodal data in videos, and thus cannot generate high-quality video description sentences. Video images and audio naturally coexist, and text can describe the fine-grained entities and actions of visual images in detail and provide semantic information of the text. By designing a hypergraph convolutional network to extract deep semantic connections between different modalities, high-quality video description sentences are generated for the video content. By utilizing the multimodal data connections of the video itself, the deep semantic connections between modalities are explicitly extracted through the hypergraph structure, with the goal of generating high-quality video description text. This solves the problem that other video description generation models do not fully utilize the multimodal data connections of videos.

[0013] Furthermore, the video multimodal data includes video data, query text data and audio data.

[0014] Further, the large model multimodal feature extraction network includes a large model EVA-CLIP visual feature extractor, an EVA-CLIP first text encoder, a large model Whisper audio encoder and a second text encoder, wherein:

[0015] The large model EVA-CLIP visual feature extractor extracts visual features of the video data;

[0016] The large model EVA-CLIP first text encoder extracts text features of the query text data;

[0017] The large model Whisper audio encoder encodes the audio data, and maps the encoded audio data into text features through the second text encoder, and embeds them into corresponding video frames.

[0018] Furthermore, the extracted multimodal features are fused to obtain fused multimodal features, including:

[0019] The visual features of the video data are multiplied by the text features of the query text data and then fused with the text features of the audio data to obtain fused multimodal features.

[0020] Furthermore, the method of using a preset hypergraph convolutional network to perform hypergraph modeling on the fused multimodal features includes:

[0021] Let any vertex be a feature of the multimodal features, and the hyperedge represent the association between the features. Use the K-nearest neighbor hypergraph modeling strategy to connect the k nearest neighbor vertices of each vertex through hyperedges to form a hypergraph structure.

[0022] Furthermore, the time division network includes a main event location module and a time division module, wherein:

[0023] The main event location module is used to predict the start time and end time of the main event matching the query text data;

[0024] The time division module is used to predict the event boundary of each sub-event in the main event.

[0025] Furthermore, the main event location module includes two detection heads, which respectively predict the start time and end time of the main event matching the query text data.

[0026] Furthermore, in the step of generating a video description sentence for each divided event using a preset video description generation network, the method further includes sampling the output of the moment division network and selecting several frames thereof as inputs of the preset video description generation network.

[0027] A second aspect of the present invention provides a video description generation system based on a hypergraph convolutional network, comprising:

[0028] A multimodal feature extraction module, wherein the multimodal feature extraction module uses a preset large-model multimodal feature extraction network to extract multimodal features of video multimodal data;

[0029] A multimodal feature fusion module, wherein the multimodal feature fusion module fuses the extracted multimodal features to obtain fused multimodal features;

[0030] A hypergraph modeling module, wherein the hypergraph modeling module uses a preset hypergraph convolutional network to perform hypergraph modeling on the fused multimodal features to obtain a hypergraph;

[0031] An event output module, wherein the event output module processes the hypergraph using a preset time division network, deletes video content irrelevant to the query text, and divides relevant video content into events;

[0032] The video description generation module generates a video description sentence for each divided event using a preset video description generation network.

[0033] The third aspect of the present invention provides a computer medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating video description based on a hypergraph convolutional network is implemented.

[0034] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0035] The present invention aims to solve the problem that the existing methods are insufficient in the ability to extract multimodal features of videos, cannot capture the deep semantic information of multimodal features, and thus cannot generate high-quality video descriptions. The present invention adopts a video description generation method based on a hypergraph convolutional network. The hypergraph convolutional network constructed by the present invention utilizes the powerful feature extraction capability of a large model in the multimodal feature extraction stage without introducing too many operations, extracts visual, textual and audio features from the video respectively, enhances the density of semantic information connections between different modalities in the multimodal data fusion stage, obtains deep semantic connections in multimodal data, and enables the model to capture more fine-grained semantic information, thereby generating high-quality video description text. The generated video description text increased by 0.64 in the indicator METEOR, 1.09 in the indicator CIDEr, and 2.75 in the indicator SPICE. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a process of generating a video description based on a hypergraph convolutional network provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of data flow of a method for generating video description based on a hypergraph convolutional network provided in an embodiment of the present invention;

[0038] Figure 3 A flowchart of a method for generating video description based on a hypergraph convolutional network provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the experimental effect of a video description generation method based on a hypergraph convolutional network provided by an embodiment of the present invention;

[0040] Figure 5 A module schematic diagram of a video description generation system based on a hypergraph convolutional network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0042] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0043] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0044] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0045] Example 1

[0046] The embodiment of the present invention provides a method for generating video description based on a hypergraph convolutional network. Figures 1 to 3 As shown, the following steps are included:

[0047] The multimodal features of the video multimodal data are extracted using a preset large-model multimodal feature extraction network;

[0048] The extracted multimodal features are fused to obtain fused multimodal features;

[0049] Using a preset hypergraph convolutional network to perform hypergraph modeling on the fused multimodal features to obtain a hypergraph;

[0050] Processing the hypergraph using a preset time division network, deleting video content irrelevant to the query text and dividing relevant video content into events;

[0051] The preset video description generation network is used to generate a video description sentence for each divided event.

[0052] In a further embodiment, the video multimodal data includes video data, query text data and audio data.

[0053] In a further embodiment, the large model multimodal feature extraction network includes a large model EVA-CLIP visual feature extractor, a large model EVA-CLIP first text encoder, a large model Whisper audio encoder and a second text encoder, wherein:

[0054] The large model EVA-CLIP visual feature extractor extracts visual features of the video data;

[0055] In a specific embodiment, the visual features of the video data are extracted by a large model EVA-CLIP visual feature extractor;

[0056] The large model EVA-CLIP first text encoder extracts text features of the query text data;

[0057] In a specific embodiment, extracting text features of the query text data by a large model EVA-CLIP text encoder;

[0058] The large model Whisper audio encoder encodes the audio data, and maps the encoded audio data into text features through the second text encoder, and embeds the text features into corresponding video frames;

[0059] In a specific embodiment, after the audio data is processed by the large model Whisper, the transcribed audio code is output, and then the transcribed audio code is mapped into a text embedding using the MiniLM text encoder and embedded into each video frame.

[0060] In a further embodiment, the step of fusing the extracted multimodal features to obtain fused multimodal features includes:

[0061] For multimodal information fusion, dimensional addition or multiplication operation is an intuitive and simple method and is widely used. Therefore, the visual features of the video data are multiplied by the text features of the query text data and then fused with the text features of the audio data to obtain the fused multimodal features.

[0062] In a further embodiment, the method of performing hypergraph modeling on the fused multimodal features using a preset hypergraph convolutional network includes:

[0063] First, the K-nearest neighbor strategy is used to model the hypergraph structure of multimodal data, and then the spectral domain-based hypergraph convolution is used to perform deep semantic extraction on the multimodal data; let any vertex be a feature of the multimodal features, and the hyperedge represents the relationship between the features. Using the K-nearest neighbor hypergraph modeling strategy, the k nearest neighbor vertices of each vertex are connected by hyperedges to form a hypergraph structure.

[0064] In order to reduce the introduction of redundant or even noisy information when constructing a hypergraph, which may cause confusion in the model and increase the complexity of learning, the embodiment of the present invention does not adopt a construction strategy in which each vertex is connected to other vertices on the hypergraph through hyperedges to construct a dense hypergraph. Instead, the k nearest neighbors of each vertex are connected through hyperedges to construct a relatively sparse hypergraph structure.

[0065] In a further embodiment, in order to ensure that the data received by the video description generation network is related to the main content of the video, generate high-quality video description text, build a time division network, delete video content that is not related to the query text, and divide the relevant video content into events, the time division network includes a main event positioning module and a time division module, wherein:

[0066] The main event location module is used to predict the start time and end time of the main event matching the query text data;

[0067] The time division module aggregates multimodal information in the video to increase the information difference between different sub-events, thereby more accurately predicting the event boundary of each sub-event.

[0068] In a further embodiment, the main event location module includes two detection heads, which respectively predict the start time and the end time of the main event matching the query text data.

[0069] In a further embodiment, the step of generating a video description sentence for each divided event using a preset video description generation network also includes sampling the output of the moment division network and selecting 20 frames thereof as input of the preset video description generation network.

[0070] In a further embodiment, a cross-attention mechanism of an autoregressive text decoder is used to focus on the output of a multimodal encoder to generate high-quality video description text.

[0071] In a specific embodiment, Figure 4 As shown in Figure 2, the video description generation method based on the hypergraph convolutional network performs multimodal deep semantic extraction through the hypergraph, which can capture the fine-grained information such as actions and entities of different sub-events, thereby generating high-quality video description text, such as Figure 4 As shown. It clearly demonstrates the outstanding performance of the model in locating the main events, moment segmentation and video description on the video content of "How to make a plastic bag explode". The model not only successfully retrieved the video moments (34-108 seconds) that matched the manual annotations, but also the predicted event boundaries and event descriptions maintained a high degree of semantic correspondence with the labels and video content. For example, the predicted description "add water to it" (34-45s) matches the description of the label. "Take the required materials" (49-52s) refers to vinegar. This fully demonstrates the ability of the model method of the present invention to understand and analyze such challenging tasks, as well as its potential value in practical applications.

[0072] Example 2

[0073] This embodiment provides a video description generation system based on a hypergraph convolutional network. Figure 5 As shown, including:

[0074] A multimodal feature extraction module, wherein the multimodal feature extraction module uses a preset large-model multimodal feature extraction network to extract multimodal features of video multimodal data;

[0075] A multimodal feature fusion module, wherein the multimodal feature fusion module fuses the extracted multimodal features to obtain fused multimodal features;

[0076] A hypergraph modeling module, wherein the hypergraph modeling module uses a preset hypergraph convolutional network to perform hypergraph modeling on the fused multimodal features to obtain a hypergraph;

[0077] An event output module, wherein the event output module processes the hypergraph using a preset time division network, deletes video content irrelevant to the query text, and divides relevant video content into events;

[0078] The video description generation module generates a video description sentence for each divided event using a preset video description generation network.

[0079] Example 3

[0080] This embodiment provides a computer medium, characterized in that a computer program is stored on the computer medium, and when the computer program is executed by a processor, the video description generation method based on the hypergraph convolutional network described in Example 1 is implemented.

[0081] The same or similar reference numerals correspond to the same or similar components;

[0082] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0083] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A video description generation method based on a hypergraph convolutional network, characterized in that: The following steps are involved: The multimodal features of the video multimodal data are extracted using a preset large-model multimodal feature extraction network; The extracted multimodal features are fused to obtain fused multimodal features; Using a preset hypergraph convolutional network to perform hypergraph modeling on the fused multimodal features to obtain a hypergraph; Processing the hypergraph using a preset time division network, deleting video content irrelevant to the query text and dividing relevant video content into events; The preset video description generation network is used to generate a video description sentence for each divided event.

2. The method for generating video description based on a hypergraph convolutional network according to claim 1, characterized in that: The video multimodal data includes video data, query text data and audio data.

3. The method for generating video description based on a hypergraph convolutional network according to claim 2, characterized in that: The large model multimodal feature extraction network includes a large model EVA-CLIP visual feature extractor, a large model EVA-CLIP first text encoder, a large model Whisper audio encoder, and a second text encoder, wherein: The large model EVA-CLIP visual feature extractor extracts visual features of the video data; The large model EVA-CLIP first text encoder extracts text features of the query text data; The large model Whisper audio encoder encodes the audio data, and maps the encoded audio data into text features through the second text encoder, and embeds them into corresponding video frames.

4. The method for generating video description based on a hypergraph convolutional network according to claim 3, characterized in that: The step of fusing the extracted multimodal features to obtain fused multimodal features includes: The visual features of the video data are multiplied by the text features of the query text data and then fused with the text features of the audio data to obtain fused multimodal features.

5. The method for generating video description based on a hypergraph convolutional network according to claim 1, characterized in that: The method of using a preset hypergraph convolutional network to perform hypergraph modeling on the fused multimodal features includes: Let any vertex be a feature of the multimodal features, and the hyperedge represent the association between the features. Use the K-nearest neighbor hypergraph modeling strategy to connect the k nearest neighbor vertices of each vertex through hyperedges to form a hypergraph structure.

6. The method for generating video description based on a hypergraph convolutional network according to any one of claims 2 to 5, characterized in that: The time division network includes a main event location module and a time division module, wherein: The main event location module is used to predict the start time and end time of the main event matching the query text data; The time division module is used to predict the event boundary of each sub-event in the main event.

7. The method for generating video description based on a hypergraph convolutional network according to claim 6, characterized in that: The main event location module includes two detection heads, which respectively predict the start time and the end time of the main event matching the query text data.

8. The method for generating video description based on a hypergraph convolutional network according to claim 6, characterized in that: The step of generating a video description sentence for each divided event using a preset video description generation network also includes sampling the output of the time division network and selecting several frames therein as inputs of the preset video description generation network.

9. A video description generation system based on a hypergraph convolutional network, characterized in that: include: A multimodal feature extraction module, wherein the multimodal feature extraction module uses a preset large-model multimodal feature extraction network to extract multimodal features of video multimodal data; A multimodal feature fusion module, wherein the multimodal feature fusion module fuses the extracted multimodal features to obtain fused multimodal features; A hypergraph modeling module, wherein the hypergraph modeling module uses a preset hypergraph convolutional network to perform hypergraph modeling on the fused multimodal features to obtain a hypergraph; An event output module, wherein the event output module processes the hypergraph using a preset time division network, deletes video content irrelevant to the query text, and divides relevant video content into events; The video description generation module generates a video description sentence for each divided event using a preset video description generation network.

10. A computer medium, characterized in that The computer medium stores a computer program, and when the computer program is executed by the processor, the method for generating video description based on a hypergraph convolutional network according to any one of claims 1 to 8 is implemented.

Images