Video information summary generation method and apparatus, and electronic apparatus and storage medium

Abstract

The present application relates to a video information summary generation method and apparatus, and an electronic apparatus and a storage medium. The video information summary generation method comprises: acquiring an initial video and a preset text description for a target object; inputting the initial video and the text description into a trained open-set object detection model for key frame detection, so as to obtain key frames including the target object in the initial video; performing clustering on several key frames to obtain an initial video information summary; inputting the initial video information summary into a picture-text extraction unit for text description extraction, so as to obtain a picture-text description of the initial video information summary; inputting the picture-text description of the initial video information summary and the initial video information summary into a video-text semantic alignment unit for semantic alignment, so as to obtain aligned video feature representations; and inputting the aligned video feature representations into a text generation unit, so as to obtain a target video information summary. Therefore, the correctness of the content of a text summary is improved.

Description

Video information summarization methods, apparatus, electronic devices, and storage media Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to video information summarization methods, apparatus, electronic devices, and storage media. Background Technology

[0002] In video surveillance scenarios such as security, transportation, kitchens, and schools, a large amount of video data is generated. This video data is often quite long and contains a lot of redundant and invalid information. Directly watching these long videos to obtain useful information is very time-consuming and inefficient.

[0003] Video summarization technology primarily extracts and generates keyframes from videos automatically or semi-automatically, helping users quickly understand the video content, especially key information in long videos, without needing to watch the entire video. Traditional static video summarization techniques mainly rely on keyframe representation and clustering methods to extract representative keyframes from the video and combine them into a new video to achieve video summarization. With the development of deep learning, some new methods for extracting keyframes have emerged. For example, using a CNN model to extract spatial features from video frames and then using an LSTM model to process temporal features, the model learns to identify important feature change points and predict keyframes; or using object detection algorithms to identify keyframes containing the main object. However, static video summarization techniques based on traditional object detection algorithms can only detect objects that the trained model can recognize and generate video summaries. To apply this technology in new scenarios, it is necessary to collect relevant data to retrain the object detection model, which is time-consuming and laborious, and limits the flexibility of the technology's application. By identifying key objects through object detection algorithms, video frames containing key objects are used as keyframes to compose video summaries. The video summaries obtained in this way may contain multiple events. This type of video summary increases the difficulty of video understanding and makes it impossible to generate a text summary that accurately represents the video content.

[0004] There is currently no effective solution to the problem that related technologies cannot generate text summaries that accurately represent video content. Summary of the Invention

[0005] This embodiment provides a video information summarization method, apparatus, electronic device, and storage medium to solve the problem in related technologies that it is impossible to generate text summaries that correctly represent video content.

[0006] Firstly, this embodiment provides a video information summary generation method, including:

[0007] Obtain the initial video and a preset text description for the target object;

[0008] The initial video and text description are input into the trained open-world object detection model to perform keyframe detection, and the keyframes containing the target object in the initial video are obtained.

[0009] Clustering of key frames from several frames yields an initial video information summary.

[0010] The initial video information summary is input into the image-text extraction unit to extract the text description, resulting in the image text description of the initial video information summary;

[0011] The image text description of the initial video information summary and the initial video information summary are input into the video-text semantic alignment unit for semantic alignment to obtain the aligned video feature representation;

[0012] The aligned video feature representation is input into the text generation unit to obtain the target video information summary.

[0013] In some embodiments, keyframes from several frames are clustered to obtain an initial video information digest, including:

[0014] Extract feature representations from keyframes across several frames;

[0015] The feature representations of keyframes are input into the trained backbone network model. Based on the backbone network model, K-means clustering analysis is performed to obtain the initial video information summary.

[0016] In some of these embodiments, the backbone network model employs the Swing Transformer model.

[0017] In some embodiments, K-means clustering analysis is performed based on the modular backbone network to obtain an initial video information summary, including:

[0018] Based on the modular backbone network, the cosine similarity between keyframes is used as a distance metric. K-means clustering analysis is then performed based on the distance metric to obtain an initial video information summary.

[0019] In some embodiments, the feature representations of keyframes are input into the trained module backbone network. Based on the module backbone network, K-means clustering analysis is performed to obtain an initial video information summary, including:

[0020] The feature representation of keyframes containing the target object is input into the module backbone network, and K-means clustering analysis is performed based on the module backbone network to obtain multiple initial video information summaries.

[0021] The K-nearest neighbor method is used to detect abnormal frames in the initial video information digest of the clustering, and video frames with a value greater than a preset threshold are removed to obtain the initial video information digest.

[0022] In some embodiments, the initial video information digest is input to the image-text extraction unit to obtain an image-text description of the initial video information digest, including:

[0023] Feature representations of keyframes in the initial video information summary are extracted using the pre-trained BLIP-2 model;

[0024] The feature representations of keyframes in the initial video information summary are input into an autoregressive text generator to generate text, resulting in an image text description of the initial video information summary.

[0025] In some of these embodiments, the text generation unit is an autoregressive text generator.

[0026] Secondly, this embodiment provides a video information summarization generation device, including: an acquisition module, a keyframe extraction module, a keyframe clustering module, an image-to-text conversion module, an alignment module, and a generation module, wherein:

[0027] The acquisition module is used to acquire the initial video and a preset text description of the target object;

[0028] The keyframe extraction module is used to input the initial video and text description into the trained open-world object detection model to perform keyframe detection and obtain keyframes containing the target object in the initial video.

[0029] The keyframe clustering module is used to cluster key frames from a number of frames to obtain an initial video information summary;

[0030] The image-to-text conversion module is used to input the initial video information summary into the image-to-text extraction unit to extract the text description, and obtain the image-to-text description of the initial video information summary;

[0031] The alignment module is used to input the image text description of the initial video information summary and the initial video information summary into the video-text semantic alignment unit for semantic alignment, so as to obtain the aligned video feature representation;

[0032] The generation module is used to input the aligned video feature representation into the text generation unit to obtain the target video information summary.

[0033] Thirdly, this embodiment provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video information summary generation method described in the first aspect above.

[0034] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the video information summary generation method described in the first aspect above.

[0035] Compared with related technologies, the video information summary generation method provided in this embodiment obtains an initial video and a preset text description of the target object; inputs the initial video and text description into a trained open-world object detection model for keyframe detection to obtain keyframes containing the target object in the initial video; clusters the keyframes of several frames to obtain an initial video information summary; inputs the initial video information summary into an image-text extraction unit to extract the text description, obtaining an image-text description of the initial video information summary; inputs the image-text description of the initial video information summary and the initial video information summary into a video-text semantic alignment unit for semantic alignment to obtain an aligned video feature representation; and inputs the aligned video feature representation into a text generation unit to obtain the target video information summary. This achieves the generation of a text summary that correctly describes the video content, improving the accuracy of the text summary content.

[0036] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0038] Figure 1 is a hardware structure block diagram of the terminal of the video information digest generation method of this embodiment.

[0039] Figure 2 is a flowchart of the video information summary generation method in this embodiment.

[0040] Figure 3 is a flowchart of another video information summary generation method in this embodiment.

[0041] Figure 4 is a preferred flowchart of the video information summary generation method of this embodiment.

[0042] Figure 5 is a structural block diagram of the video information summary generation device in this embodiment. Detailed Implementation

[0043] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0044] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0045] The method embodiments provided in this example can be executed in a terminal, computer, or similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of the terminal for the video information summary generation method of this embodiment. As shown in Figure 1, the terminal may include one or more processors 102 (only one is shown in Figure 1) and a memory 104 for storing data, wherein the processor 102 may be, but is not limited to, a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that the structure shown in Figure 1 is merely illustrative and does not limit the structure of the terminal. For example, the terminal may include more or fewer components than shown in Figure 1, or have a different configuration than that shown in Figure 1.

[0046] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the video information summary generation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0048] This embodiment provides a video information digest generation method. Figure 2 is a flowchart of the video information digest generation method of this embodiment. As shown in Figure 2, the process includes the following steps:

[0049] Step S201: Obtain the initial video and the preset text description for the target object.

[0050] Specifically, the initial video is acquired. This initial video can come from various sources, such as surveillance cameras, social media platforms, personal mobile phones, and professional shooting equipment. These videos may contain different resolutions, frame rates, and encoding formats. The initial video content can include various elements, such as people, animals, scenes, and objects. The specific initial video selected can be chosen based on actual application needs. This embodiment does not impose specific limitations on this. A text description of the target object is pre-defined as needed. This text description is used to extract the required video summary. The text description can include the target object's name, attributes, location, behavior, etc.

[0051] Step S202: Input the initial video and text description into the trained open-world object detection model to perform keyframe detection, and obtain keyframes in the initial video that contain the target object.

[0052] Traditional object detection algorithms (close-set object detection) are limited to predefined known categories, i.e., categories that have appeared in the training set. They cannot identify targets when faced with new scenes or labels. For example, if the model is trained with the label "lion," it will identify all targets labeled "lion," but it cannot detect horses or more precise targets, such as only identifying the lion on the left side of the image. Furthermore, most traditional object detection models are supervised models, requiring the collection of a large amount of labeled data for training. Each time the set of identifiable objects needs to be expanded or changed, data must be collected and labeled, which is time-consuming, labor-intensive, and lacks flexibility. This application uses an open-world object detection model. The open-world object detection model is pre-trained to obtain a trained model. Pre-defined text descriptions of target objects and initial video are input into the trained open-world object detection model for keyframe detection, resulting in keyframes containing the target object. The open-world object detection model can include multiple target events, improving object detection capability and flexibility.

[0053] Step S203: Cluster the key frames of several frames to obtain an initial video information summary.

[0054] Specifically, the keyframes obtained from the textual descriptions of the preset target objects include multiple preset target objects. When aggregating the keyframes, analysis can be performed by identifying the scenes, actions, and events corresponding to different target objects in the keyframes, or by analyzing the colors, textures, and shapes of different target objects in the keyframes, or by analyzing the keyframes based on image quality, content similarity, and action changes, or by analyzing different target objects in the keyframes based on semantic space relevance. After analysis, the main content, structure, and temporal relationships of different target objects in the keyframes can be understood. By aggregating keyframes with the same target objects in several keyframes, a coherent video summary containing multiple aggregates of different target objects can be obtained, which is the initial video information summary.

[0055] Step S204: Input the initial video information summary into the image-text extraction unit to extract the text description, and obtain the image text description of the initial video information summary.

[0056] The initial video information summary consists of a series of keyframes, which represent the main content of the video. Before being input into the image-to-text extraction unit, the keyframes can undergo some preprocessing, including scaling, cropping, and denoising, to improve the accuracy of subsequent image recognition and text extraction.

[0057] Before extracting the text description, the image-to-text extraction unit loads the model, algorithm, and parameters. The model can be a pre-trained image recognition model, object detection model, OCR (Optical Character Recognition) model, or natural language processing model, etc.

[0058] The image-to-text extraction unit first performs image content recognition on each keyframe in the initial video information summary. This includes recognizing objects, scenes, actions, etc., in the image. By utilizing a pre-trained image recognition model, the image-to-text extraction unit identifies the main content in the keyframes and converts it into image-text descriptions.

[0059] Step S205: Input the image text description of the initial video information summary and the initial video information summary into the video-text semantic alignment unit for semantic alignment to obtain the aligned video feature representation.

[0060] Specifically, the image text description contains key information and descriptions extracted from keyframes, while the initial video information summary contains the main content and structure of the video. The video-text semantic alignment unit semantically aligns the information in the image text description with the information in the initial video information summary to obtain the corresponding video feature representation. Specific alignment methods can include rule-based matching, statistical model-based matching, deep learning-based matching, etc. The specific alignment method can be determined according to the actual situation, and this application provides specific limitations on this. Through alignment, each part of the image text description is matched with the corresponding content in the initial video information summary, ensuring their semantic consistency and enhancing the feature representation of the video summary.

[0061] Step S206: Input the aligned video feature representation into the text generation unit to obtain the target video information summary.

[0062] Specifically, the text generation unit is a module that converts the input data into a target video information summary. The aligned video feature representation is input into the text generation unit, which converts the aligned video feature representation into the corresponding text description to obtain the target video information summary.

[0063] Through steps S201 to S206, an initial video and a preset text description of the target object are obtained. The initial video and text description are input into a trained open-world object detection model for keyframe detection to obtain keyframes containing the target object in the initial video. The keyframes of several frames are aggregated to obtain an initial video information summary. The initial video information summary is input into an image-text extraction unit to extract text descriptions to obtain image text descriptions of the initial video information summary. The image text descriptions of the initial video information summary and the initial video information summary are input into a video-text semantic alignment unit for semantic alignment to obtain aligned video feature representations. The aligned video feature representations are input into a text generation unit to obtain the target video information summary. Compared with the current traditional static video summarization technology that extracts representative keyframes from the video to form a new video summary, this application uses an open-world object detection model to extract keyframes from the initial video based on the preset text description of the target object, then aggregates the keyframes to obtain video information summaries of multiple target objects, then extracts image text descriptions from the video information summaries of multiple target objects, aligns the image text descriptions with the initial video information summary information, and converts them into text output to obtain the target video information summary. It enables the generation of text summaries that accurately represent video content, thereby improving the accuracy of the text summaries.

[0064] In some embodiments, keyframes from several frames are clustered to obtain an initial video information summary, including: extracting feature representations of the keyframes from several frames; inputting the feature representations of the keyframes into a trained backbone network model; and performing K-means clustering analysis based on the backbone network model to obtain the initial video information summary. Specifically, based on the Swin Transformer model used in the backbone network model, the cosine similarity between keyframes is used as a distance metric, and K-means clustering analysis is performed based on this distance metric to obtain the initial video information summary.

[0065] Specifically, different backbone network models have different representation capabilities. This invention uses the Swin Transformer model as the backbone network model. Through hierarchical feature representation and shifted window computation, the model has stronger modeling capabilities at different scales and linear computational complexity, enhancing performance and accelerating computation. Since there may be multiple target objects in the preset text description of the target object, keyframes with the same features are clustered according to different target objects or needs to obtain initial video information summaries for different target objects. Through the embedding module, feature representations of several keyframes are extracted and input into the trained Swin Transformer backbone network model. The cosine similarity between keyframes is calculated, and K-means clustering is performed using the cosine similarity as a distance metric to obtain N / M (rounded up, where N is the number of video frames after clustering and M is the number of frames included in the initial video summary) clusters, which is the initial video information summary.

[0066] In another embodiment, the feature representations of keyframes are input into the trained module backbone network. Based on the module backbone network, K-means clustering analysis is performed to obtain an initial video information summary, including:

[0067] The feature representation of keyframes containing the target object is input into the module backbone network. K-means clustering analysis is performed based on the module backbone network to obtain multiple initial video information summaries. The K-nearest neighbor method is used to detect abnormal frames in the initial video information summaries and remove video frames that exceed a preset threshold to obtain the initial video information summaries.

[0068] Specifically, keyframes are clustered based on features to obtain an initial video information digest, which may contain anomalous frames. For anomalous frames, an anomalous frame detection module is set up to detect them. This embodiment uses the K-nearest neighbor method to detect anomalous frames in the initial video information digest, specifically including:

[0069] (1) While acquiring keyframes, record the frame number of the keyframes and arrange the keyframes in the initial video information summary of the cluster in ascending order according to the frame number;

[0070] (2) Set the number of nearest neighbor points k = 5;

[0071] (3) Calculate the average distance between each frame number and its k nearest points;

[0072] (4) Remove keyframes whose average distance is greater than the threshold (set to 5 in this embodiment). These keyframes are abnormal frames.

[0073] In some embodiments, the initial video information summary is input to the image-text extraction unit to obtain an image-text description of the initial video information summary, including: extracting feature representations of key frames in the initial video information summary using a pre-trained BLIP-2 model; and inputting the feature representations of key frames in the initial video information summary to an autoregressive text generator to generate text, thereby obtaining an image-text description of the initial video information summary.

[0074] Specifically, the BLIP-2 (Bidirectional Language-Image Pre-training with Pairings 2) model is used to extract feature representations of keyframes from the initial video information summary. The BLIP-2 model can process both image and text data simultaneously. Through its visual-language interaction capabilities, it identifies keyframes in the video and then transforms these keyframes into feature representations. These feature representations contain the visual information of the keyframes, providing a foundation for subsequent text generation. The extracted keyframe feature representations are then input into an autoregressive text generator, which uses an autoregressive approach to generate text descriptions, resulting in the image-text description of the initial video information summary.

[0075] In some embodiments, the aligned video feature representation is input to a text generation unit to obtain a target video information summary, wherein the text generation unit is an autoregressive text generator.

[0076] Specifically, the text generation unit employs an autoregressive text generator, which takes the aligned video feature representation as input and generates the corresponding target video information summary based on the prediction and autoregressive function of the autoregressive text generator.

[0077] This embodiment also provides a video information digest generation method. Figure 3 is a flowchart of another video information digest generation method according to this embodiment. As shown in Figure 3, the process includes the following steps:

[0078] Step S301: Extract text descriptions using the image-text extraction unit;

[0079] Specifically, the clustered videos in the initial video information summary obtained by clustering are used as input, the feature representations of keyframes in each video are extracted, and the feature representations are input into the sub-regression text generator to generate text descriptions of the keyframes.

[0080] Step S302: The text description information is semantically aligned with the initial video information summary through the video-text semantic alignment unit to obtain the aligned video feature representation.

[0081] Specifically, textual description information and initial video information summaries are input into the video-text semantic alignment unit. Features from the textual description information are matched with features from the initial video information to obtain aligned video feature representations, ensuring semantic consistency and enhancing the feature representation of the video summary. Within the video-text semantic alignment unit, the visual embedder is built based on the Video Swing Transformer, and the text embedder is built based on the CLIP Text Encoder.

[0082] Step S303: The aligned video feature representation is input into the autoregressive text generator through the text generation unit to obtain the target video information summary.

[0083] Specifically, the text generation unit employs an autoregressive text generator, using aligned video feature representations as input. Based on the predictions and autoregressive function of the autoregressive text generator, it generates corresponding target video information summaries. This autoregressive text generator is constructed using a Transformer encoder structure.

[0084] Through steps S301 to S303, the image-text extraction unit obtains the text description of each frame in the initial video information summary; the video-text semantic alignment unit enhances the feature representation of the initial video information summary with the semantic information of the text description of each frame; finally, the text generation unit obtains the target video information summary using an autoregressive approach. The video information summary generation algorithm architecture proposed in this embodiment can reduce the semantic gap between video and text, map visual and linguistic representations to a shared semantic space, improve video description capabilities, and enhance the accuracy of the text summary content.

[0085] Figure 4 is a preferred flowchart of the video information digest generation method of this embodiment. As shown in Figure 4, the video information digest generation method includes the following steps:

[0086] Step S401: Obtain the initial video and a preset text description for the target object;

[0087] Step S402: Input the initial video and text description into the trained open-world object detection model to perform keyframe detection, and obtain keyframes in the initial video that contain the target object;

[0088] Step S403: Extract feature representations of keyframes from several frames;

[0089] Step S404: Input the feature representation of the keyframe into the trained Swin Transformer backbone network model, use the cosine similarity between keyframes as the distance metric, perform K-means clustering analysis, and obtain multiple initial video information summaries for clustering.

[0090] Step S405: Use the K-nearest neighbor method to detect abnormal frames in the initial video information digest of the clustering, remove video frames that are greater than a preset threshold, and obtain the initial video information digest.

[0091] Step S406: Extract feature representations of keyframes from the initial video information summary using the pre-trained BLIP-2 model;

[0092] Step S407: Input the feature representation of the keyframe in the initial video information summary into the image-text extraction unit to generate text, and obtain the image text description of the initial video information summary;

[0093] Step S408: Input the image text description of the initial video information summary and the initial video information summary into the video-text semantic alignment unit for semantic alignment to obtain the aligned video feature representation;

[0094] Step S409: Input the aligned video feature representation into the text generation unit to obtain the target video information summary.

[0095] This embodiment also provides a video information summarization generation device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform predetermined functions. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0096] Figure 5 is a structural block diagram of the video information summary generation device of this embodiment. As shown in Figure 5, the device 50 includes: an acquisition module 51, a keyframe extraction module 52, a keyframe clustering module 53, an image-to-text conversion module 54, an alignment module 55, and a generation module 56, wherein:

[0097] The acquisition module 51 is used to acquire the initial video and the preset text description of the target object;

[0098] The keyframe extraction module 52 is used to input the initial video and text description into the trained open-world object detection model to perform keyframe detection and obtain keyframes containing the target object in the initial video.

[0099] The keyframe clustering module 53 is used to cluster keyframes of several frames to obtain an initial video information summary.

[0100] The image-to-text conversion module 54 is used to input the initial video information summary into the image-to-text extraction unit to extract the text description and obtain the image-to-text description of the initial video information summary.

[0101] Alignment module 55 is used to input the image text description of the initial video information summary and the initial video information summary into the video-text semantic alignment unit for semantic alignment, so as to obtain the aligned video feature representation;

[0102] The generation module 56 is used to input the aligned video feature representation into the text generation unit to obtain the target video information summary.

[0103] This embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0104] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0105] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0106] S1, Obtain the initial video and preset text descriptions for the target object;

[0107] S2, input the initial video and text description into the trained open-world object detection model to perform keyframe detection, and obtain the keyframes in the initial video that contain the target object;

[0108] S3, cluster the key frames of several frames to obtain the initial video information summary;

[0109] S4, input the initial video information summary into the image-text extraction unit to extract the text description, and obtain the image text description of the initial video information summary;

[0110] S5, input the image text description of the initial video information summary and the initial video information summary into the video-text semantic alignment unit for semantic alignment, and obtain the aligned video feature representation;

[0111] S6. The aligned video feature representation is input into the text generation unit to obtain the target video information summary.

[0112] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0113] Furthermore, in conjunction with the video information digest generation method provided in the above embodiments, this embodiment can also provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the video information digest generation methods described in the above embodiments.

[0114] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0115] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0116] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0118] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for generating video information summaries, characterized in that, include: Obtain the initial video and a preset text description for the target object; The initial video and the text description are input into the trained open-world object detection model to perform keyframe detection, thereby obtaining keyframes in the initial video that contain the target object. Cluster the key frames from several frames to obtain an initial video information summary; The initial video information summary is input into the image-text extraction unit to extract the text description, thereby obtaining the image text description of the initial video information summary; The image text description of the initial video information summary and the initial video information summary are input into the video-text semantic alignment unit for semantic alignment to obtain the aligned video feature representation; The aligned video feature representation is input into the text generation unit to obtain the target video information summary.

2. The video information digest generation method according to claim 1, characterized in that, The process of clustering the keyframes across several frames to obtain an initial video information summary includes: Extract the feature representation of the key frame from the plurality of frames; The feature representations of the keyframes are input into the trained backbone network model. Based on the backbone network model, K-means clustering analysis is performed to obtain the initial video information summary.

3. The video information digest generation method according to claim 2, characterized in that, The backbone network model adopts the Swing Transformer model.

4. The video information digest generation method according to claim 2, characterized in that, The initial video information summary is obtained by performing K-means clustering analysis based on the module backbone network, including: Based on the backbone network of the module, the cosine similarity between the keyframes is used as a distance metric. K-means clustering analysis is performed based on the distance metric to obtain the initial video information summary.

5. The video information digest generation method according to claim 2, characterized in that, The feature representation of the keyframe is input into the trained module backbone network, and K-means clustering analysis is performed based on the module backbone network to obtain the initial video information summary, including: The feature representation of the key frame containing the target object is input into the module backbone network, and K-means clustering analysis is performed based on the module backbone network to obtain multiple initial video information summaries of clusters; The K-nearest neighbor method is used to detect abnormal frames in the initial video information digest of the clustering, and video frames with values ​​greater than a preset threshold are removed to obtain the initial video information digest.

6. The video information digest generation method according to claim 1, characterized in that, The step of inputting the initial video information digest into the image-text extraction unit to obtain the image text description of the initial video information digest includes: Feature representations of keyframes in the initial video information summary are extracted using the pre-trained BLIP-2 model; The feature representations of keyframes in the initial video information summary are input into an autoregressive text generator to generate text, resulting in an image text description of the initial video information summary.

7. The video information digest generation method according to claim 1, characterized in that, The text generation unit is an autoregressive text generator.

8. A video information summarization generation device, characterized in that, include: The module comprises an acquisition module, a keyframe extraction module, a keyframe clustering module, an image-to-text conversion module, an alignment module, and a generation module, among which: The acquisition module is used to acquire the initial video and a preset text description of the target object; The keyframe extraction module is used to input the initial video and the text description into the trained open-world object detection model to perform keyframe detection, and obtain keyframes in the initial video containing the target object. The keyframe clustering module is used to cluster the keyframes of several frames to obtain an initial video information summary. The image-to-text conversion module is used to input the initial video information summary into the image-to-text extraction unit to extract the text description, thereby obtaining the image-to-text description of the initial video information summary; The alignment module is used to input the image text description of the initial video information summary and the initial video information summary into the video-text semantic alignment unit for semantic alignment, so as to obtain the aligned video feature representation; The generation module is used to input the aligned video feature representation into the text generation unit to obtain the target video information summary.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the video information summary generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the video information summary generation method according to any one of claims 1 to 7.

Images