Video-oriented event knowledge extraction method, system and equipment and medium

Abstract

The invention belongs to the technical field of video processing, and discloses a video-oriented event knowledge extraction method, system and device and a medium, and the method comprises the steps: carrying out the video frame segmentation of a to-be-processed video, and carrying out the extraction of each video frame based on a physical space angle and an embedded space angle, and obtaining a plurality of key frames; generating an image title corresponding to each key frame; inputting each key frame and the corresponding image title into a multi-modal event extraction model for event extraction to obtain an event corresponding to each key frame; wherein the multi-modal event extraction model comprises a feature extraction module, a multi-modal feature fusion module based on an attention mechanism and a classifier which are connected in sequence; and based on the inherent time sequence information of the to-be-processed video, sorting and concluding the key frames and the corresponding events to obtain event knowledge and event evolution trajectories in the to-be-processed video. According to the technical scheme, the accuracy and richness of event knowledge extraction can be improved, and the necessity of manual participation is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular relates to a method, system, device and medium for extracting event knowledge from a video. Background Art

[0002] Event knowledge extraction is an important technology for information extraction. Its main task is to identify events from the corpus and extract relevant information about the events. Accurately discovering events and extracting relevant information are of great significance to event prediction, security monitoring, autonomous driving and other fields. With the continuous popularization and rapid development of mobile Internet technology and network communication technology, video data has become an important data source in the real world. Therefore, how to interpret events in videos has become a hot issue in the fields of natural language processing and computer vision.

[0003] Compared with single-modal event knowledge extraction, multi-modal event knowledge extraction has more advantages because it can integrate information from different modalities and complement each other. Video data contains a lot of visual information. The synergy of visual information and text information can provide a more comprehensive and accurate description of events. However, there are many difficulties in this task:

[0004] First, directly learning features from videos is time-consuming and computationally expensive. The amount of video data is usually very large, and processing videos requires parsing each frame of its continuous frames, which places extremely high demands on computing resources and processing time. Secondly, fine-grained annotation of videos is time-consuming and labor-intensive. Fine-grained annotation of videos not only needs to be accurate to the frame level, but also requires a detailed description of the start and end time of each event, the objects involved, and the specific content of the event. This annotation process consumes a lot of manpower and time, and in reality, most videos lack such detailed annotation data. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system, device and medium for event knowledge extraction for video to solve the problems existing in the above-mentioned prior art.

[0006] To achieve the above object, the present invention provides a method for extracting event knowledge from a video, comprising:

[0007] Get the video to be processed;

[0008] Performing video frame segmentation on the video to be processed to obtain a plurality of video frames;

[0009] Extracting each of the video frames based on the physical space angle and the embedded space angle to obtain a number of key frames;

[0010] Generate an image title corresponding to each of the key frames;

[0011] Input each of the key frames and the corresponding image title into a multimodal event extraction model for event extraction, and obtain the event corresponding to each of the key frames; wherein the multimodal event extraction model includes a feature extraction module, a multimodal feature fusion module based on an attention mechanism, and a classifier connected in sequence;

[0012] Based on the inherent time sequence information of the video to be processed, each of the key frames and the corresponding events are sorted and summarized to obtain the event knowledge and event evolution trajectory in the video to be processed.

[0013] Optionally, the video frames are extracted based on the physical space angle and the embedded space angle to obtain a number of key frames, specifically including:

[0014] Extracting a physical space key video frame based on mutual information data between every two video frames in each of the video frames;

[0015] Extracting an embedding space key video frame based on cosine similarity data between feature vectors of every two video frames in each of the video frames;

[0016] The physical space key video frame and the embedded space key video frame are used as the key frames.

[0017] Optionally, extracting a physical space key video frame based on the mutual information data between every two video frames in each of the key frames specifically includes:

[0018] Step 1: performing grayscale processing on each of the video frames;

[0019] Step 2: Calculate the mutual information data between every two video frames;

[0020] Step 3: When the calculated mutual information data is greater than a preset mutual information threshold, the latter frame of the two video frames is discarded;

[0021] Repeat steps 1 and 2 until the calculation of each of the video frames is completed, and use the remaining video frames as physical space key video frames.

[0022] Optionally, extracting the embedding space key video frame based on the cosine similarity data between the feature vectors of every two video frames in each of the key frames specifically includes:

[0023] Step 1: Input each of the video frames into the ViT-B / 16 model to extract the embedding representation, and obtain the high-dimensional embedding representation data corresponding to each of the video frames;

[0024] Step 2: Calculate the cosine similarity data between the high-dimensional embedding representation data corresponding to each two video frames. When the calculated cosine similarity data is greater than a preset cosine similarity threshold, discard the latter frame of the two video frames.

[0025] Repeat step 1 to step 2 until the calculation of each video frame is completed, and use the remaining video frames as key video frames for embedding space.

[0026] Optionally, generating an image title corresponding to each key frame specifically includes:

[0027] Acquire generated title prompt data, input each of the key frames and the generated title prompt data into an LLaVA model to capture visual information, and obtain an image description;

[0028] The image description corresponding to each of the key frames is input into the Llama model for semantic overview to obtain the image title corresponding to each of the key frames.

[0029] Optionally, the training process of the multimodal event extraction model specifically includes:

[0030] Acquire training data, wherein the training data includes training video frame data and corresponding events, and the training video frame data includes training video frames and corresponding image titles;

[0031] An initial multimodal event extraction model is constructed, the training data is input into the initial multimodal event extraction model for event extraction, and training is performed with the goal of minimizing the loss between the initial training results after event extraction and the events corresponding to the training video frame data to obtain the multimodal event extraction model.

[0032] Optionally, the processing process of the multimodal event extraction model specifically includes:

[0033] The feature extraction module includes a text feature extraction sub-model and an image feature extraction sub-model;

[0034] Extracting semantic features of the image title corresponding to the key frame through the text feature extraction sub-model to obtain text semantic representation data;

[0035] Performing image recognition on the key frame through the image feature extraction sub-model to obtain picture sequence vector representation data, and performing dimension conversion on the picture sequence vector representation data to obtain object vector representation data;

[0036] Inputting the text semantic representation data, the picture sequence vector representation data and the object vector representation data into the multimodal feature fusion module to perform multimodal feature fusion to obtain fused feature data;

[0037] The fused feature data is input into the classifier to extract events, and the events corresponding to the key frames are obtained.

[0038] A video-oriented event knowledge extraction system, comprising:

[0039] The data acquisition module is used to obtain the video to be processed; segment the video to be processed to obtain a plurality of video frames; extract each of the video frames based on the physical space angle and the embedded space angle to obtain a plurality of key frames; generate an image title corresponding to each of the key frames;

[0040] A multimodal event extraction module, used for inputting each of the key frames and the corresponding image titles into a multimodal event extraction model for event extraction, and obtaining the events corresponding to each of the key frames; wherein the multimodal event extraction model comprises a feature extraction module, a multimodal feature fusion module based on an attention mechanism, and a classifier connected in sequence;

[0041] The video event knowledge acquisition module is used to sort and summarize each key frame and the corresponding event according to the inherent time sequence information of the video to be processed, so as to obtain the event knowledge and event evolution trajectory in the video to be processed.

[0042] An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the video-oriented event knowledge extraction method.

[0043] A computer-readable storage medium stores a computer program, which implements the video-oriented event knowledge extraction method when executed by a processor.

[0044] The technical effects of the present invention are:

[0045] The present invention extracts key frames from videos by combining the physical space and embedded space of video images, reduces video-level tasks to image-level tasks, and greatly reduces computational costs while ensuring task accuracy. By selecting representative frames in the physical space and further optimizing the selection process of key frames in the embedded space, the amount of data to be processed can be effectively reduced, thereby improving processing efficiency. The present invention automatically generates text descriptions that are highly relevant to the content of key frames. This not only improves the accuracy and consistency of annotations, but also greatly reduces the need for manual participation. The model of the present invention can better understand and process the relationship between visual information and text information, thereby improving the accuracy and richness of event knowledge extraction. The present invention uses the timestamp of the video frame to restore the event knowledge of the extracted key video frame to the complete event chain and context of the entire video, and constructs the final video-oriented event knowledge extraction result. In this way, not only can independent sub-events in the video be identified, but also the association and evolution process between sub-events can be understood. In summary, the present invention realizes video-oriented event knowledge extraction through key frame extraction, automatic title generation, multimodal model fine-tuning, and timestamp context construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 is an implementation flow chart of an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of video frame title generation based on a dual-major model in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as limiting the present invention, but should be understood as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0051] It should be understood that the terms described in the present invention are only for describing special embodiments and are not intended to limit the present invention. In addition, for the numerical range in the present invention, it should be understood that each intermediate value between the upper and lower limits of the scope is also specifically disclosed. Each smaller range between the intermediate value in any stated value or stated range and any other stated value or intermediate value in the described range is also included in the present invention. The upper and lower limits of these smaller ranges can be independently included or excluded in the scope.

[0052] Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art generally understood by the present invention. Although the present invention describes only preferred methods, any method similar or equivalent to that described herein may also be used in the implementation or testing of the present invention. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of a conflict with any incorporated document, the content of this specification shall prevail.

[0053] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments of the present invention description without departing from the scope or spirit of the present invention. Other embodiments derived from the present invention description will be apparent to those skilled in the art. The present application description and examples are exemplary only.

[0054] The words “include,” “including,” “have,” “contain,” etc. used in this article are open-ended terms, meaning including but not limited to.

[0055] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0056] Embodiment 1

[0057] like Figure 1 - Figure 2 As shown, this embodiment provides a video-oriented event knowledge extraction method, including: obtaining a video to be processed; performing video frame segmentation on the video to be processed to obtain a plurality of video frames; extracting each of the video frames based on a physical space angle and an embedded space angle to obtain a plurality of key frames; generating an image title corresponding to each of the key frames; inputting each of the key frames and the corresponding image title into a multimodal event extraction model for event extraction to obtain an event corresponding to each of the key frames; wherein the multimodal event extraction model includes a feature extraction module connected in sequence, a multimodal feature fusion module based on an attention mechanism, and a classifier; based on the inherent time sequence information of the video to be processed, each of the key frames and the corresponding events are sorted and summarized to obtain event knowledge and event evolution trajectories in the video to be processed.

[0058] In this embodiment, the existing event knowledge extraction for video is improved. First, the similarity of video frames in real physical space and embedded space is used to extract key frames, and the differences of video frames are described in multiple levels; secondly, the powerful image understanding and natural language generation capabilities of the large model are used to generate titles for key frames; thirdly, a fine-grained multimodal event extraction method is proposed to integrate the global picture, local object and text information of the key frame; finally, the extracted event knowledge is summarized based on the natural time sequence of the original video.

[0059] In order to solve the deficiencies in the prior art, this embodiment proposes a method for extracting event knowledge from a video, which specifically includes the following steps:

[0060] S1: Dual keyframe extraction based on physical space and embedded space of video images;

[0061] S2: Video frame caption generation based on dual large models;

[0062] S3: Multimodal event extraction integrating fine-grained graphic and text information;

[0063] S4: Event knowledge induction based on natural temporal sequence of videos.

[0064] Step S1 measures the difference of video frames from two perspectives: physical space difference and embedded space difference of video frames, and sets thresholds to extract key frames. The proposed method specifically includes:

[0065] S101: Learning physical spatial characteristics of video frames based on mutual information;

[0066] S102: Video frame feature learning based on ViT-B / 16 model;

[0067] S103: Filter key video frames based on similarity.

[0068] The most important information in a video is the visual information. However, each video contains a large number of video frames, and there is redundancy between video frames, which brings challenges to both computational and time costs. Therefore, before extracting event knowledge from a video, it is necessary to select the key frames of the video.

[0069] The key frames of the video should be representative and thematically significant, and the redundancy of the screened video frames should be low, so as to express the entire video content. Traditional technologies mainly use methods such as color histograms, SIFT and HOG. With the development of deep learning, networks such as ResNet and LSTM are used to extract high-level semantic features of videos to help extract key frames. This embodiment combines the physical space feature learning of video frames based on mutual information and the video frame feature learning based on the ViT-B / 16 model to screen video frames, achieving the dual effects of improving accuracy and increasing interpretability.

[0070] The video frame physical space characteristic learning based on mutual information may include the following contents:

[0071] Mutual information is the relationship between the joint entropy H(X,Y) of two random variables X and Y and their respective entropies H(X) and H(Y), and is defined as follows: I(X;Y) = H(X) + H(Y) - H(X,Y).

[0072] Where H(X) is the entropy of random variable X, H(Y) is the entropy of random variable Y, and H(X,Y) is the joint entropy of X and Y. Mutual information is a measure of the amount of shared information between two images and can be used to evaluate the correlation between images. When the mutual information value of two images is low, it means that the amount of information they share is small and the correlation between the images is weak. This may mean that the two images are greatly deviated or unrelated in space.

[0073] In this embodiment, the original video is first divided into picture frames by a uniform sampling method. 1000 frames of pictures are extracted from each video to represent the entire video, and the size of each picture is 224×224.

[0074] To calculate the mutual information between each two frames, we first need to convert them into grayscale images I and J and calculate the joint histogram of the pixel values ​​of the two images:

[0075] H IJ (i,j)=number of times (I(x,y)=i and J(x,y)=j)

[0076] Then, the joint histogram is normalized to obtain the joint probability distribution:

[0077]

[0078] Where N is the total number of pixels in the image.

[0079] Then, the marginal probability distribution is calculated from the joint probability distribution:

[0080] P I (i)=∑ j P IJ(i,j)

[0081] P J (j) = ∑ i P IJ (i,j)

[0082] Finally, according to the definition formula of mutual information, the mutual information of two frames is calculated:

[0083]

[0084] The video frame feature learning based on the ViT-B / 16 model can be implemented as follows:

[0085] This embodiment uses a pre-trained model ViT-B / 16 to obtain feature information of video frames. The ViT-B / 16 model directly divides the image into patches of fixed size and inputs these patches as sequences into the Transformer model, thereby eliminating the need for convolution operations in traditional convolutional neural networks. The performance of ViT-B / 16 has been proven to exceed traditional image feature extraction methods such as Resnet, VGG, and I3D.

[0086] This embodiment also divides the original video into picture frames by uniform sampling, and extracts 100 frames of pictures from each video to represent the entire video.

[0087] First, input each frame image x∈R H×W×C Segment into non-overlapping fixed-size patches. Assuming the input image size is H×W and the size of each patch is P×P, the image is segmented into Each patch is flattened into a vector, so the dimensions of the patch are P×P×C.

[0088] Project each patch vector to a fixed dimension D through a linear layer:

[0089]

[0090] in, is the linear embedding matrix; E pos ∈R N×D is the position embedding matrix.

[0091] Similar to the [CLS] tag in BERT, ViT uses an additional classification tag:

[0092]

[0093] in is a learnable categorical label embedding.

[0094] The ViT-B / 16 model contains L layers of Transformers, each of which includes a multi-head self-attention mechanism and a feedforward neural network. Each feedforward neural network usually consists of two linear layers and an activation function:

[0095] FFN(z)=ReLU(zW1+b1)W2+b2

[0096] Each Transformer layer also includes layer normalization and skip connection (Residual Connection):

[0097] Z l =LN(MHSA(Z l-1 )+Z l-1 )

[0098] Z l =LN(FFN(Z l ')+Z l ')

[0099] So far, this embodiment has learned the embedding of each frame in the high-dimensional space through the ViT-B / 16 model for subsequent screening.

[0100] It is feasible that the similarity-based key video frame screening mainly involves selecting video frames using the learned mutual information (physical space) and feature values ​​(embedded space) between image pairs.

[0101] First, for the similarity measurement of the physical space, this embodiment sets a threshold for the mutual information between images. When the result is higher than the threshold, the latter frame of the two video frames is discarded.

[0102] Secondly, for the similarity measurement of the embedding space, this embodiment adopts cosine similarity calculation.

[0103] Cosine Similarity is a measure used to measure the directional similarity of two non-zero vectors in a multidimensional space. It is defined as the ratio of the dot product of two vectors to the product of their modulus lengths:

[0104]

[0105] Among them, A·B is the dot product of A and B, ||A|| and ||B|| are the norms of vectors A and B respectively, A i and B i are the i-th components of vectors A and B respectively.

[0106] The closer the result is to 1, the higher the similarity between vectors A and B, that is, the higher the similarity between the two frames. Then, a threshold is set for the cosine similarity, and when the result is higher than the threshold, the latter frame of the two video frames is discarded.

[0107] Finally, by combining the above two methods, this embodiment realizes the extraction of key frames in the video based on the physical space and embedded space of the video picture.

[0108] The video frame title generation based on the dual-major model can be implemented as follows: the task of image title generation is to generate a text describing the input image, specifically including:

[0109] This embodiment combines LLaVA (Large Language-and-Vision Assistant) and Llama3 to implement the task of generating titles for video frames. LLaVA is a multimodal model based on the combination of a large language model and a visual model, which aims to provide more advanced understanding and processing capabilities for visual tasks. LLaVA combines the powerful capabilities of natural language processing with the sophisticated analysis of computer vision, can fuse text and visual information, extract semantic information from images and videos, and generate natural language descriptions. Llama is a large language model based on the Transformer architecture developed by Meta, focusing on natural language processing tasks. Therefore, LLaVA focuses on the fusion application of visual information and text information, while Llama focuses on text processing. This embodiment makes full use of the characteristics of LLaVA and Llama, and designs a pipeline working principle to generate titles for video frames.

[0110] Specifically, this embodiment first guides LLaVA to generate an image description with the help of prompts, and then the description is used as the input of Llama3 to generate an image title.

[0111] For the generation of image descriptions, the following prompts are designed:

[0112] “What do you see in this image?”

[0113] For image caption generation, the following prompts are designed:

[0114] Please generate a captionfor this image using this description:<image description> )”

[0115] Thus, this embodiment has expanded the text information for the video frame by combining two large language models, converted the unimodal task into a multimodal task, and provided data for subsequent processing.

[0116] The multimodal event extraction integrating fine-grained graphic and text information can be implemented as follows:

[0117] S301: Event semantic learning based on BERT;

[0118] S302: Image feature learning based on YOLO;

[0119] S303: Multimodal feature fusion based on attention mechanism;

[0120] S304: Outputting the extraction results based on the classifier.

[0121] Previous research on event knowledge extraction has mainly focused on a single modality, such as text data, and there have been fewer studies on multimodal event knowledge extraction. In addition, in research on multimodal event extraction, the processing of visual information mostly remains at a coarse-grained level. Previous technical methods directly learned the embedding of text and its corresponding pictures, and then fused them, ignoring the processing of fine-grained information. For example, in an image, global information and objects in the image can be supplemented as fine-grained local information. This part of information usually has a direct correspondence with event arguments. Therefore, this embodiment proposes multimodal event extraction that integrates fine-grained graphic and text information.

[0122] The event semantic learning based on BERT may include the following contents: BERT (Bidirectional Encoder Representations from Transformers) is a language representation model released by Google AI in 2018. Its core idea is to use the bidirectional encoder of Transformer to capture the contextual information of the text. In this embodiment, BERT is first used to extract text semantic features, and then the extracted features are processed using a dynamic multi-pooling mechanism to obtain more representative features.

[0123] Let W = (w1, w2, ..., w i ,…,w n ) represents the input text sequence, where n represents the length of the text sequence, and w i (1≤i≤n) represents the i-th word in the text sequence. This example follows the original BERT and uses the [CLS] and [SEP] special identifiers. The final input is:

[0124] [CLS],w1,w2,…,w i ,…,w n,[SEP]

[0125] After BERT encoding, the vector representation is obtained:

[0126] X=(x0,x1,x2,…,x i ,…,x n ,x n+1 )

[0127] x0 and x n+1 are the vector representations of [CLS] and [SEP], respectively, X=(x1,x2,…,x i ,…,x n ) is the vector of text sequence. i (1≤i≤n) represents the vector representation of the i-th word in the text sequence.

[0128] The YOLO-based image feature learning includes the following contents: learning the global (entire picture) features and local (object) features of the key frames, specifically: this embodiment still extracts the global features of the image based on Vi TB / 16. In addition, for fine-grained local features, the YOLO model is fine-tuned to identify fine-grained object information in the picture. First, the input image is divided into S×S grids. Each grid is responsible for predicting the position and category of one or more objects.

[0129] Each bounding box in the grid is characterized by the coordinates of the center of the bounding box (x, y) and the width and height of the bounding box relative to the entire image (w, h).

[0130] For each grid cell, predict B bounding boxes and their confidence scores:

[0131]

[0132] Among them, P(Object) represents the probability that an object exists in the grid cell. It represents the intersection-over-union ratio of the predicted bounding box and the true bounding box.

[0133] In addition to the bounding box information, each grid also predicts the conditional probability P (Class i ∣Object), where Class i is the i-th category.

[0134] For each grid cell, the final prediction is the bounding box location and size (x, y, w, h) and the category confidence score:

[0135] Class Confidence=P(Class i ∣Object)×Confidence

[0136] The loss function consists of three parts: coordinate loss, confidence loss, and category loss. Coordinate loss and confidence loss use mean square error, and category loss uses cross entropy loss.

[0137] So far, for the input video frame, YOLO is used to identify the object in the picture and obtain its vector representation, and then the fully connected layer is used to transform the dimension of the vector representation to obtain the object vector representation O = (o1, o2, ..., o m ). Where m is the number of objects recognized.

[0138] The multimodal feature fusion based on the attention mechanism can be implemented as follows:

[0139] The input is the text semantic representation X, the image sequence vector representation M and the object vector representation O extracted by the first two modules, and the output is the image features and object features obtained through the cross-attention mechanism. The basic idea of ​​the cross-attention mechanism is to use the representation of one feature (query, Query) to pay attention to the representation of another feature (key, Key and value, Value).

[0140] First, for the text semantic representation X and the image sequence vector M, we can define:

[0141]

[0142] in, and are all learnable weight matrices.

[0143] Similarly, for the object vector representation O in the video frame, we can define:

[0144]

[0145] in, is the learnable weight matrix.

[0146] The attention scores for the image sequence vector M and the object vector representation O are calculated as follows:

[0147]

[0148] Then, the attention weights are used to calculate the weights for the image sequence vector M and the object vector O.

[0149] Z M =A M V M

[0150] Z O =A O V O

[0151] Finally, the text features, video frame global features, and video frame local features are fused using the following formula:

[0152] F=W F [Z M ; Z O ]+b F

[0153] Among them, W F is the learnable weight matrix, b F is the bias, and [;] represents the concatenation operation of two vectors.

[0154] The classifier-based extraction result output may include the following:

[0155] Input the multimodal feature vector into the classifier for classification:

[0156] y i =softmax(W s F m +b s )

[0157] Among them, W s and b s is a trainable parameter.

[0158] Using cross entropy as the loss function:

[0159]

[0160] in, is the true value, y i is the predicted value.

[0161] The event knowledge induction based on the natural time sequence of the video may include the following contents:

[0162] This embodiment uses the inherent time sequence information in the video data to summarize the extracted event knowledge. By sorting the timestamps of key video frames, the events occurring in the video can be summarized in time sequence, and the event knowledge covered in the video and the evolution trajectory of the events can be explicitly presented. Through the timestamp information, the event knowledge of discrete video frames is reorganized into a sequence, representing the event knowledge of the original video, thereby restoring the task to the video itself. So far, this embodiment has completed all the steps of video-oriented event knowledge extraction.

[0163] Compared with the prior art, the advantages and positive effects of this embodiment are:

[0164] First, this embodiment extracts key frames from the video by combining the physical space and embedding space of the video screen, reducing the video-level task to the image-level task, and greatly reducing the computational cost while ensuring the task accuracy. By selecting representative frames in the physical space and further optimizing the key frame selection process in the embedding space, the amount of data to be processed can be effectively reduced, thereby improving the processing efficiency. Second, this embodiment uses a large model to generate high-quality titles for key video frames, reducing the burden of manual annotation. Through the pre-trained natural language processing model, a text description that is highly relevant to the key frame content is automatically generated. This not only improves the accuracy and consistency of annotation, but also greatly reduces the need for manual participation. Third, this embodiment enables the model to better understand and process the relationship between visual information and text information through fine-tuning on the task data set, thereby improving the accuracy and richness of event knowledge extraction. Fourth, this embodiment uses the timestamp of the video frame to restore the event knowledge of the extracted key video frame to the complete event chain and context of the entire video, and constructs the final video-oriented event knowledge extraction result. In this way, not only can the independent sub-events in the video be identified, but also the association and evolution process between the sub-events can be understood. In summary, this embodiment realizes event knowledge extraction for video through key frame extraction, automatic title generation, multimodal model fine-tuning and timestamp context construction.

[0165] refer to Figure 1 This embodiment designs a method for extracting event knowledge from video, which can be specifically divided into the following four steps: dual keyframe extraction based on the physical space and embedded space of the video screen (S1), video frame title generation based on dual large models (S2), multimodal event extraction integrating fine-grained graphic information (S3), and event knowledge induction based on the natural time sequence of the video (S4). S1 to S4 are independent of each other. The keyframe output of S1 is the input of S2 and S3, the keyframe title output of S2 is the input of S3, and the event knowledge extracted by S3 is the input of S4.

[0166] It is feasible that S1 includes three core steps: learning the physical space characteristics of video frames based on mutual information (S101), learning the video frame features based on the ViT-B / 16 model (S102), and screening key video frames based on similarity (S103). S3 includes four core steps: event semantic learning based on BERT (S301), image feature learning based on YOLO (S302), multimodal feature fusion based on attention mechanism (S303), and output of extraction results based on classifier (S304). S101 and S102 are in a parallel relationship, and the outputs of both are the inputs of S103; S301 and S302 are also in a parallel relationship, and the outputs of both are the inputs of S303, and the output of S303 is the input of S304.

[0167] refer to Figure 2 For S2, this embodiment designs a video frame title generation scheme based on the dual large models of LLaVA and Llama to generate descriptive text and titles for the input video frames. By designing prompts, the large model is guided to produce high-quality natural language. Since LLaVA is better at image understanding, this embodiment uses LLaVA to generate descriptions of key video frames. For example, a video frame is first given, and then the question is asked: "What do you see in this picture?" Then, LLaVA gives the descriptive text of the video frame: "It is a sunny day in the picture. There is a little girl on the lawn. The little girl is flying a kite." Since Llama is better at natural language understanding, this embodiment uses Llama to summarize the descriptive text produced by LLaVA to generate the title of the video frame. As shown in the prompt below the dotted line: "Please use this description to generate a title for this image: It is a sunny day in the picture. There is a little girl on the lawn. The little girl is flying a kite." Llama summarizes the description and gives the title of the final video frame: "It is a sunny day, and the little girl is flying a kite on the lawn."

[0168] Therefore, this embodiment combines the training characteristics of the two large models, gives full play to the strengths of both, and designs a pipeline prompt learning to expand the text information for the video frame, transforming the single-modal task into a multi-modal task.

[0169] A video-oriented event knowledge extraction system, comprising:

[0170] The data acquisition module is used to obtain the video to be processed; segment the video to be processed to obtain a plurality of video frames; extract each of the video frames based on the physical space angle and the embedded space angle to obtain a plurality of key frames; generate an image title corresponding to each of the key frames;

[0171] A multimodal event extraction module, used for inputting each of the key frames and the corresponding image titles into a multimodal event extraction model for event extraction, and obtaining the events corresponding to each of the key frames; wherein the multimodal event extraction model comprises a feature extraction module, a multimodal feature fusion module based on an attention mechanism, and a classifier connected in sequence;

[0172] The video event knowledge acquisition module is used to sort and summarize each key frame and the corresponding event according to the inherent time sequence information of the video to be processed, so as to obtain the event knowledge and event evolution trajectory in the video to be processed.

[0173] An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the video-oriented event knowledge extraction method.

[0174] A computer-readable storage medium stores a computer program, which implements the video-oriented event knowledge extraction method when executed by a processor.

[0175] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video-based event knowledge extraction method, characterized in that: include: Get the video to be processed; Performing video frame segmentation on the video to be processed to obtain a plurality of video frames; Extracting each of the video frames based on the physical space angle and the embedded space angle to obtain a number of key frames; Generate an image title corresponding to each of the key frames; Input each of the key frames and the corresponding image title into a multimodal event extraction model for event extraction, and obtain the event corresponding to each of the key frames; wherein the multimodal event extraction model includes a feature extraction module, a multimodal feature fusion module based on an attention mechanism, and a classifier connected in sequence; Based on the inherent time sequence information of the video to be processed, each of the key frames and the corresponding events are sorted and summarized to obtain the event knowledge and event evolution trajectory in the video to be processed.

2. The method for extracting event knowledge from video according to claim 1, characterized in that: The video frames are extracted based on the physical space angle and the embedded space angle to obtain several key frames, including: Extracting a physical space key video frame based on mutual information data between every two video frames in each of the video frames; Extracting an embedding space key video frame based on cosine similarity data between feature vectors of every two video frames in each of the video frames; The physical space key video frame and the embedded space key video frame are used as the key frames.

3. The method for extracting event knowledge from video according to claim 2, characterized in that: Extracting a physical space key video frame based on the mutual information data between every two video frames in each of the key frames specifically includes: Step 1: performing grayscale processing on each of the video frames; Step 2: Calculate the mutual information data between every two video frames; Step 3: When the calculated mutual information data is greater than a preset mutual information threshold, the latter frame of the two video frames is discarded; Repeat steps 1 and 2 until the calculation of each of the video frames is completed, and use the remaining video frames as physical space key video frames.

4. The method for extracting event knowledge from video according to claim 2, characterized in that: Extracting the embedding space key video frame based on the cosine similarity data between the feature vectors of every two video frames in each of the key frames specifically includes: Step 1: Input each of the video frames into the ViT-B / 16 model to extract the embedding representation, and obtain the high-dimensional embedding representation data corresponding to each of the video frames; Step 2: Calculate the cosine similarity data between the high-dimensional embedding representation data corresponding to each two video frames. When the calculated cosine similarity data is greater than a preset cosine similarity threshold, discard the latter frame of the two video frames. Repeat step 1 to step 2 until the calculation of each video frame is completed, and use the remaining video frames as key video frames for embedding space.

5. The method for extracting event knowledge from video according to claim 1, characterized in that: Generating an image title corresponding to each of the key frames specifically includes: Acquire generated title prompt data, input each of the key frames and the generated title prompt data into an LLaVA model to capture visual information, and obtain an image description; The image description corresponding to each of the key frames is input into the Llama model for semantic summarization to obtain the image title corresponding to each of the key frames.

6. The method for extracting event knowledge from video according to claim 1, characterized in that: The training process of the multimodal event extraction model specifically includes: Acquire training data, wherein the training data includes training video frame data and corresponding events, and the training video frame data includes training video frames and corresponding image titles; An initial multimodal event extraction model is constructed, the training data is input into the initial multimodal event extraction model for event extraction, and training is performed with the goal of minimizing the loss between the initial training results after event extraction and the events corresponding to the training video frame data to obtain the multimodal event extraction model.

7. The method for extracting event knowledge from video according to claim 1, characterized in that: The processing process of the multimodal event extraction model specifically includes: The feature extraction module includes a text feature extraction sub-model and an image feature extraction sub-model; Extracting semantic features of the image title corresponding to the key frame through the text feature extraction sub-model to obtain text semantic representation data; Performing image recognition on the key frame through the image feature extraction sub-model to obtain picture sequence vector representation data, and performing dimension conversion on the picture sequence vector representation data to obtain object vector representation data; Inputting the text semantic representation data, the picture sequence vector representation data and the object vector representation data into the multimodal feature fusion module to perform multimodal feature fusion to obtain fused feature data; The fused feature data is input into the classifier to extract events, and the events corresponding to the key frames are obtained.

8. A video-oriented event knowledge extraction system, characterized in that: include: A data acquisition module, used to obtain the video to be processed; Performing video frame segmentation on the video to be processed to obtain a plurality of video frames; Extracting each of the video frames based on the physical space angle and the embedded space angle to obtain a number of key frames; generating an image title corresponding to each of the key frames; A multimodal event extraction module, used for inputting each of the key frames and the corresponding image titles into a multimodal event extraction model for event extraction, and obtaining the events corresponding to each of the key frames; wherein the multimodal event extraction model comprises a feature extraction module, a multimodal feature fusion module based on an attention mechanism, and a classifier connected in sequence; The video event knowledge acquisition module is used to sort and summarize each key frame and the corresponding event according to the inherent time sequence information of the video to be processed, so as to obtain the event knowledge and event evolution trajectory in the video to be processed.

9. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute a video-oriented event knowledge extraction method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements a video-oriented event knowledge extraction method as described in any one of claims 1-7.

Images