Video clip summary generation method and system based on knowledge enhancement

Abstract

The present invention belongs to the field of data processing technology. A method and system for generating video clip summaries based on knowledge enhancement is provided, which extracts information of multiple video frames of a video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set, and further obtains a first common sense graph with common sense relationships between objects as edges, a second common sense graph with scene relationships between objects as edges, and a third common sense graph with spatiotemporal relationships between objects as edges; after integrating the first common sense graph, the second common sense graph, and the third common sense graph, a graph attention network is used to obtain representations of all objects, and the representations of all objects and the vector representation set are spliced ​​into a video representation, and the video representation and the prompt word text are used as inputs of a large language model to obtain a summary text description of the video clip; the present invention improves the accuracy and comprehensiveness of video summary generation by integrating common sense knowledge, scene knowledge, and spatiotemporal knowledge.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for generating video clip summaries based on knowledge enhancement. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] With the explosive growth of video content, how to efficiently generate video clip summaries to quickly capture key information in the video has become a technical problem that needs to be solved urgently.

[0004] Currently, video summary generation technology mainly relies on direct analysis of video content, but the effect is not ideal when dealing with complex scenes, implicit information and long time series relationships; traditional computer vision-based methods can only understand the objects and characters appearing in the video at the perceptual level, but cannot understand the relationship between them; moreover, existing knowledge enhancement technology mainly focuses on common sense knowledge, and enhances the performance of video understanding models by introducing common sense knowledge in knowledge graphs. Although it solves the problem of lack of visual common sense in video understanding models to a certain extent, it often focuses on the association and expansion of a single concept, and lacks in-depth mining of the overall context and implicit information of the video. Summary of the invention

[0005] In order to address the deficiencies of the prior art, the present invention provides a method and system for generating video clip summaries based on knowledge enhancement, which improves the accuracy and comprehensiveness of video summary generation by integrating common sense knowledge, scene knowledge and spatiotemporal knowledge.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for generating video clip summaries based on knowledge enhancement.

[0008] A video clip summary generation method based on knowledge enhancement includes the following processes:

[0009] Extracting information of multiple video frames of the video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set;

[0010] According to the object name set, knowledge triples with the same name are retrieved in the common sense knowledge graph to obtain the first common sense graph with object names as nodes and common sense relationships between objects as edges;

[0011] Extract visual relationship triplets from the visual representation set to obtain a second common sense graph with object names as nodes and scene relationships between objects as edges;

[0012] Extract visual relationship triplets from the visual representation set to obtain a third common sense graph with object names as nodes and spatiotemporal relationships between objects as edges;

[0013] After integrating the first common sense graph, the second common sense graph and the third common sense graph, a graph attention network is used to obtain representations of all objects, and the representations of all objects and the vector representation set are spliced ​​into a video representation. The video representation and the prompt word text are used as inputs of the large language model to obtain a summary text description of the video clip.

[0014] As a further limitation of the first aspect of the present invention, extracting information of multiple video frames of a video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set includes:

[0015] For a given video clip, uniformly extract video frames;

[0016] Extraction A collection of vector representations of video frames ,in is the representation dimension, For the t A vector representation of a video frame;

[0017] Extraction The set of object names in the video frame , Representative t The first video frame i The object names of the objects are extracted The set of visual representations in the video frame , Representative t The first video frame i The visual representation of an object is extracted The collection of text representations in the video frames , Representative t The first video frame i The text representation of objects, where For the The number of objects present in the frame.

[0018] As a further limitation of the first aspect of the present invention, according to the object name set, the knowledge triples with the same name are searched in the common sense knowledge graph to obtain a first common sense graph with the object names as nodes and the common sense relationships between the objects as edges. ,in, A collection of object names A collection of relevant common sense relations;

[0019] Extract visual relationship triplets from the visual representation set to obtain a second common sense graph with object names as nodes and scene relationships between objects as edges. ,in A collection of object names A collection of related scene relationships;

[0020] Extract visual relationship triplets from the visual representation set to obtain a third common sense graph with object names as nodes and spatiotemporal relationships between objects as edges. ,in A collection of object names A collection of related scene relationships;

[0021] Integrate the first common sense map, the second common sense map and the third common sense map into a total common sense map , = .

[0022] As a further limitation of the first aspect of the present invention, a graph attention network is used to obtain an object representation, including:

[0023] ;

[0024] in, For the t The first video frame i An object representation of an object.

[0025] As a further limitation of the first aspect of the present invention, all objects are characterized by and video vector representation Splicing into video representation , with the prompt word text Feed into a large language model to generate text description , ,in, For large language models.

[0026] In a second aspect, the present invention provides a video clip summary generation system based on knowledge enhancement.

[0027] A video clip summary generation system based on knowledge enhancement, comprising:

[0028] The video information extraction unit is configured to: extract information of multiple video frames of the video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set;

[0029] The common sense knowledge extraction unit is configured to: retrieve knowledge triples with the same name in the common sense knowledge graph according to the object name set, and obtain a first common sense graph with the object names as nodes and the common sense relationships between the objects as edges;

[0030] The scene knowledge extraction unit is configured to: extract visual relationship triples for the visual representation set to obtain a second common sense graph with object names as nodes and scene relationships between objects as edges;

[0031] The spatiotemporal knowledge extraction unit is configured to: extract visual relationship triplets for the visual representation set to obtain a third common sense graph with object names as nodes and spatiotemporal relationships between objects as edges;

[0032] The text description generation unit is configured to: integrate the first common sense graph, the second common sense graph and the third common sense graph, and then use the graph attention network to obtain the representation of all objects, splice the representation of all objects and the vector representation set into a video representation, and use the video representation and the prompt word text as the input of the large language model to obtain a summary text description of the video clip.

[0033] As a further limitation of the second aspect of the present invention, the video information extraction unit includes:

[0034] For a given video clip, uniformly extract video frames;

[0035] Extraction A collection of vector representations of video frames ,in is the representation dimension, is the vector representation of the tth video frame;

[0036] Extraction The set of object names in the video frame , Representative t The first video frame i The object names of the objects are extracted The set of visual representations in the video frame , Representative t The first video frame i The visual representation of an object is extracted The collection of text representations in the video frames , Representative t The first video frame i The text representation of objects, where For the The number of objects present in the frame.

[0037] In a third aspect, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium;

[0038] a processor adapted to execute a computer program;

[0039] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the method for generating a video clip summary based on knowledge enhancement as described in the first aspect of the present invention is implemented.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing the method for generating a video clip summary based on knowledge enhancement as described in the first aspect of the present invention.

[0041] In a fifth aspect, the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the method for generating a video clip summary based on knowledge enhancement as described in the first aspect of the present invention.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention improves the accuracy of video scene understanding by introducing scene knowledge, optimizing scene knowledge modeling, and accurately capturing the complex relationship between visual objects.

[0044] 2. The present invention introduces spatiotemporal knowledge, strengthens the integration of spatiotemporal knowledge, and comprehensively analyzes the spatiotemporal relationship of objects between video frames to ensure the timing and dynamics of video summaries.

[0045] 3. The present invention integrates common sense, scene and spatiotemporal knowledge, overcomes the shortcomings of existing solutions in terms of depth and breadth of video understanding, and provides a more efficient and accurate technical solution for video clip summary generation.

[0046] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0048] Figure 1 A schematic diagram of a process of generating a video clip summary based on knowledge enhancement provided in Embodiment 1 of the present invention;

[0049] Figure 2 A schematic diagram of generating a video clip summary using a cooking video as an example provided in Example 1 of the present invention;

[0050] Figure 3 A schematic diagram of a video clip summary generation system based on knowledge enhancement provided in Embodiment 2 of the present invention;

[0051] Figure 4 A schematic diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0054] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0055] Embodiment 1:

[0056] This implementation proposes a video clip summary generation method based on knowledge enhancement. By building a comprehensive knowledge fusion framework, it organically combines common sense knowledge, scene knowledge and spatiotemporal knowledge to achieve a comprehensive and in-depth understanding of the video content. The following is a brief introduction to the technical terms and related concepts involved in this processing solution, including:

[0057] Knowledge: a text triple consisting of "subject-predicate-object";

[0058] Common sense knowledge: knowledge related to common sense in life, such as "apple is a fruit";

[0059] Scene knowledge: the relationship between different objects in a certain frame of the video, such as "apple-on-table";

[0060] Spatiotemporal knowledge: the relationship between objects in different frames of a video, such as “person-cutting-apple”.

[0061] Specifically, the video clip summary generation method proposed in this implementation is as follows: Figure 1 As shown, the following process is included:

[0062] S1: Given a video, uniformly extract frame.

[0063] S2: Extract video information using pre-trained models.

[0064] S2.1: Use any video-language pre-trained model to extract this A collection of vector representations of frames ,in It is the representation dimension.

[0065] S2.2: Use the pre-trained target detection model to extract The set of names of all objects appearing in the frame , Representative t The first video frame i Object names, visual representations of objects , Representative t The first video frame i A visual representation of an object, a collection of textual representations , Representative t The first video frame i The text representation of objects, where For the The number of objects present in the frame.

[0066] S3: Extract common sense knowledge, using the set of object names obtained in step S2 Retrieve knowledge triplets with the same name in the common sense knowledge graph. These knowledge will form a common sense graph with object names as nodes and common sense relationships between objects as edges. ,in For object collection A collection of relevant common-sense relations.

[0067] S4: Extract scene knowledge, using the scene graph generation algorithm, for the set of visual representations extracted in step S2 Extract visual relationship triplets to form a common sense graph with object names as nodes and scene relationships between objects as edges ,in For object collection A collection of related scene relationships.

[0068] S5: Extract spatiotemporal knowledge and use the spatiotemporal scene graph generation algorithm to extract the visual representation set extracted in step S2 Extract visual relationship triplets to form a common sense graph with object names as nodes and spatiotemporal relationships between objects as edges ,in For object collection A collection of related scene relationships.

[0069] S6: Knowledge reasoning, integrating all knowledge into a large graph ,in = , using the Graph Attention Network (GAT) to get the representation of all objects ,in, For the t The first video frame i An object representation of an object.

[0070] S7: Description generation, characterizing all objects and video vector representation Splicing into video representation , with the prompt word text Send to Large Language Model (LLM) to generate text description .

[0071] like Figure 2 As shown, a cooking video is used as an example to illustrate a method for generating a video segment summary, wherein the video frame includes "oil", "egg", "pepper" and "chili";

[0072] Through scene knowledge extraction, we get: "pepper, located in the dish", "oil, located in the pot", "eggs, used for cooking", "pepper, attribute, green", "pepper, can be combined with powder";

[0073] Through spatiotemporal knowledge extraction, we get: “eggs, poured into, pot”, “peppers, poured into, pot”, “peppers, attribute, red”, and “peppers, can be combined with, powder”;

[0074] Common sense retrieval is performed through the knowledge graph, where a schematic diagram of the knowledge graph includes elements such as pepper, onion, red, spicy, noodles, cooking, powder, black, pepper, vegetable, oil, green and egg, and the relationships between the elements include "can be combined", "has...attribute", "contains", "used for", etc.;

[0075] The results of common sense screening, scene knowledge extraction, and spatiotemporal knowledge extraction are integrated through the graph attention network to obtain the representation of all objects. The representation of all objects and the video vector representation are spliced ​​into a video representation. The video representation is processed by a multi-layer perceptron (MLP) and compared with the prompt word text. Feed it into a large language model and get the result “Pour oil, eggs, pepper and chili powder into the pot and stir.”

[0076] Embodiment 2:

[0077] like Figure 3 As shown, this implementation provides a video clip summary generation system based on knowledge enhancement, including:

[0078] The video information extraction unit is configured to extract information of multiple video frames of a video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set.

[0079] More specifically, the video information extraction unit includes:

[0080] Given a video, uniformly extract Frames, use any video-language pre-trained model to extract this A collection of vector representations of frames ,in It is the representational dimension;

[0081] Using the pre-trained target detection model, extract The set of names of all objects appearing in the frame , Representative t The first video frame i Object names, visual representations of objects , Representative t The first video frame i A visual representation of an object, a collection of textual representations , Representative t The first video frame i The text representation of objects, where For the The number of objects present in the frame.

[0082] The common sense knowledge extraction unit is configured to: retrieve knowledge triples with the same name in the common sense knowledge graph according to the object name set, and obtain a first common sense graph with object names as nodes and common sense relationships between objects as edges. ,in, A collection of object names A collection of relevant common sense relations;

[0083] The scene knowledge extraction unit is configured to: extract visual relationship triplets from the visual representation set to obtain a second common sense graph with object names as nodes and scene relationships between objects as edges ,in For object collection A collection of related scene relationships;

[0084] The spatiotemporal knowledge extraction unit is configured to: extract visual relationship triplets from the visual representation set to obtain a third common sense graph with object names as nodes and spatiotemporal relationships between objects as edges ,in For object collection A collection of related scene relationships;

[0085] The text description generating unit is configured to: integrate the first common sense graph, the second common sense graph and the third common sense graph ,in = , using the graph attention network, we get the representation of all objects, specifically, including: ,in, For the t The first video frame i object representation of an object;

[0086] Characterize all objects and video vector representation Splicing into video representation . With the prompt word text Send to Large Language Model (LLM) to generate text description ,in, For large language models.

[0087] It is understandable that the above-mentioned units can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the video clip summary generation system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0088] According to another embodiment of the present application, the system described in this embodiment can be constructed and the video clip summary generation method of the embodiment of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0089] Embodiment 3:

[0090] like Figure 4 As shown, this implementation provides an electronic device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.

[0091] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0092] The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0093] The processor 1001 is configured to execute the following process:

[0094] Extracting information of multiple video frames of the video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set;

[0095] According to the object name set, knowledge triples with the same name are retrieved in the common sense knowledge graph to obtain the first common sense graph with object names as nodes and common sense relationships between objects as edges;

[0096] Extract visual relationship triplets from the visual representation set to obtain a second common sense graph with object names as nodes and scene relationships between objects as edges;

[0097] Extract visual relationship triplets from the visual representation set to obtain a third common sense graph with object names as nodes and spatiotemporal relationships between objects as edges;

[0098] After integrating the first common sense graph, the second common sense graph and the third common sense graph, a graph attention network is used to obtain representations of all objects, and the representations of all objects and the vector representation set are spliced ​​into a video representation. The video representation and the prompt word text are used as inputs of the large language model to obtain a summary text description of the video clip.

[0099] The detailed steps are described in Example 1 and will not be repeated here.

[0100] Embodiment 4:

[0101] This implementation provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0102] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0103] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the following process:

[0104] Extracting information of multiple video frames of the video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set;

[0105] According to the object name set, knowledge triples with the same name are retrieved in the common sense knowledge graph to obtain the first common sense graph with object names as nodes and common sense relationships between objects as edges;

[0106] Extract visual relationship triplets from the visual representation set to obtain a second common sense graph with object names as nodes and scene relationships between objects as edges;

[0107] Extract visual relationship triplets from the visual representation set to obtain a third common sense graph with object names as nodes and spatiotemporal relationships between objects as edges;

[0108] After integrating the first common sense graph, the second common sense graph and the third common sense graph, a graph attention network is used to obtain representations of all objects, and the representations of all objects and the vector representation set are spliced ​​into a video representation. The video representation and the prompt word text are used as inputs of the large language model to obtain a summary text description of the video clip.

[0109] The detailed steps are described in Example 1 and will not be repeated here.

[0110] Embodiment 5:

[0111] The present implementation provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the following process:

[0112] Extracting information of multiple video frames of the video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set;

[0113] According to the object name set, knowledge triples with the same name are retrieved in the common sense knowledge graph to obtain the first common sense graph with object names as nodes and common sense relationships between objects as edges;

[0114] Extract visual relationship triplets from the visual representation set to obtain a second common sense graph with object names as nodes and scene relationships between objects as edges;

[0115] Extract visual relationship triplets from the visual representation set to obtain a third common sense graph with object names as nodes and spatiotemporal relationships between objects as edges;

[0116] After integrating the first common sense graph, the second common sense graph and the third common sense graph, a graph attention network is used to obtain representations of all objects, and the representations of all objects and the vector representation set are spliced ​​into a video representation. The video representation and the prompt word text are used as inputs of the large language model to obtain a summary text description of the video clip.

[0117] The detailed steps are described in Example 1 and will not be repeated here.

[0118] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0119] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0120] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video clip summary generation method based on knowledge enhancement, characterized in that: The process includes: Extract information of multiple video frames of a video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set, including: For a given video clip, uniformly extract video frames; Extraction A collection of vector representations of video frames ,in is the representation dimension, is the vector representation of the t-th video frame, represents the set of real numbers; Extraction The set of object names in the video frame , Representative t The first video frame i The object names of the objects are extracted The set of visual representations in the video frame , Representative t The first video frame i The visual representation of an object is extracted The collection of text representations in the video frames , Representative t The first video frame i The text representation of objects, For the The number of objects present in the frame; According to the object name set, the knowledge triples with the same name are retrieved in the common sense knowledge graph to obtain the first common sense graph with object names as nodes and common sense relationships between objects as edges. ,in, A collection of object names A collection of relevant common sense relations; Extract visual relationship triplets from the visual representation set to obtain a second common sense graph with object names as nodes and scene relationships between objects as edges. ,in A collection of object names A collection of related scene relationships; Extract visual relationship triplets from the visual representation set to obtain a third common sense graph with object names as nodes and spatiotemporal relationships between objects as edges. ,in A collection of object names A collection of related scene relationships; Integrate the first common sense map, the second common sense map and the third common sense map into a total common sense map , = , using the graph attention network, we get representations of all objects, including: ； in, For the t The first video frame i object representation of an object; Characterize all objects and video vector representation Splicing into video representation , with the prompt word text Feed into a large language model to generate text description , ,in, For large language models.

2. A video clip summary generation system based on knowledge enhancement, characterized in that: include: The video information extraction unit is configured to extract information of multiple video frames of a video clip to obtain a vector representation set, an object name set, a visual representation set, and a text representation set, including: For a given video clip, uniformly extract video frames; Extraction A collection of vector representations of video frames ,in is the representation dimension, is the vector representation of the t-th video frame, represents the set of real numbers; Extraction The set of object names in the video frame , Representative t The first video frame i The object names of the objects are extracted The set of visual representations in the video frame , Representative t The first video frame i The visual representation of an object is extracted The collection of text representations in the video frames , Representative t The first video frame i The text representation of objects, For the The number of objects present in the frame; The common sense knowledge extraction unit is configured to: retrieve knowledge triples with the same name in the common sense knowledge graph according to the object name set, and obtain a first common sense graph with object names as nodes and common sense relationships between objects as edges. ,in, A collection of object names A collection of relevant common sense relations; The scene knowledge extraction unit is configured to: extract visual relationship triplets from the visual representation set to obtain a second common sense graph with object names as nodes and scene relationships between objects as edges ,in A collection of object names A collection of related scene relationships; The spatiotemporal knowledge extraction unit is configured to: extract visual relationship triplets from the visual representation set to obtain a third common sense graph with object names as nodes and spatiotemporal relationships between objects as edges ,in A collection of object names A collection of related scene relationships; The text description generation unit is configured as follows: Integrate the first common sense map, the second common sense map and the third common sense map into a total common sense map , = , using the graph attention network, we get representations of all objects, including: ； in, For the t The first video frame i object representation of an object; Characterize all objects and video vector representation Splicing into video representation , with the prompt word text Feed into a large language model to generate text description , ,in, For large language models.

3. A computer device, characterized in that: include: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the method for generating a video clip summary based on knowledge enhancement as claimed in claim 1 is implemented.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method for generating a video segment summary based on knowledge enhancement as claimed in claim 1.

5. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method for generating a video segment summary based on knowledge enhancement according to claim 1 is implemented.

Images