Sample construction method and device, video processing method and device, model training method and device, equipment and medium
Patent Information
- Application Number
- CN202310630064.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
Smart Images

Figure CN119068277A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video understanding technology, and in particular to a sample construction, video processing, model training method, device, equipment and medium. The present invention can be used in video search, smart home, online education and other fields. Background Art
[0002] Video understanding is a topic that has attracted much attention from both academia and industry in the current field of computer vision research. With the rapid development of the video industry, how to use artificial intelligence technology to better understand video content is becoming increasingly important.
[0003] In the related art, there are video action recognition datasets for training video understanding models, such as the Movienet dataset. MovieNet includes 1.1 million video clips from 1,100 movies, covering various genres and languages. However, the annotation information of the datasets in the related art needs to be improved. Summary of the invention
[0004] The embodiments of this specification are intended to solve at least one of the technical problems in the related art to a certain extent. To this end, the embodiments of this specification propose a sample construction, video processing, model training method, device, equipment and medium.
[0005] The present specification provides a method for constructing a data set sample, the method comprising:
[0006] Acquire an image-caption combination corresponding to a video file; wherein the video file is divided into a plurality of unit segments; the image-caption combination is obtained based on an image frame in the video file and / or a caption text aligned with the image frame in time;
[0007] Segmenting the video file to obtain a video segment including at least one unit segment; wherein the video segment corresponds to a target image-caption combination; and the unit segment to which the target image frame included in the target image-caption combination belongs is the target unit segment;
[0008] Generating hierarchical marking information of the target image-subtitle combination based on the segment identifier of the target unit segment; wherein the hierarchical marking information is used to represent the hierarchical structure of the target image-subtitle combination in the video file;
[0009] The dataset sample is constructed based on the hierarchical tag information of the combination of the video clip and the target image caption.
[0010] In one embodiment, the unit segment is a scene, and the unit segment to which the target image frame included in the target image-caption combination belongs is the target scene; the hierarchical marking information of the target image-caption combination is generated based on the segment identifier of the target unit segment, including
[0011] The hierarchical marking information of the target image-caption combination is generated based on the scene identification of the target scene.
[0012] In one embodiment, the unit segment is a storyboard, and the unit segment to which the target image frame included in the target image-subtitle combination belongs is the target storyboard; the hierarchical marking information of the target image-subtitle combination is generated based on the segment identifier of the target unit segment, including
[0013] The hierarchical marking information of the target image-caption combination is generated based on the storyboard identifier of the target storyboard.
[0014] In one embodiment, the unit segment includes a scene and a storyboard, the storyboard to which the target image frame included in the target image-caption combination belongs is the target storyboard, and the scene to which the target storyboard belongs is the target scene; and the hierarchical marking information of the target image-caption combination is generated based on the segment identifier of the target unit segment, including:
[0015] The hierarchical marking information of the target image-caption combination is generated based on the storyboard identifier of the target storyboard and the scene identifier of the target scene.
[0016] In one embodiment, the video file includes a first scene, and the method further includes:
[0017] Determining an initial scene boundary frame of the first scene and a storyboard boundary frame of the storyboard;
[0018] If the initial scene boundary frame of the first scene is not a storyboard boundary frame of any storyboard, determining the any storyboard as a divided storyboard;
[0019] Boundary alignment processing is performed based on the storyboard boundary frames of the divided storyboards and the initial scene boundary frame of the first scene to obtain a target scene boundary frame of the first scene; wherein the target scene boundary frame is used as a reference for segmenting the video file.
[0020] In one embodiment, the video file includes a second scene adjacent to the first scene; the boundary alignment process is performed based on the storyboard boundary frames of the divided storyboards and the initial scene boundary frame of the first scene to obtain a target scene boundary frame of the first scene;
[0021] Determine a first matching condition between the divided storyboard and the first scene, and a second matching condition between the divided storyboard and the second scene;
[0022] Determining, according to the first matching situation and the second matching situation, a target matching scene matching the divided storyboard in the first scene and the second scene;
[0023] The target scene boundary frame is set according to the storyboard boundary frame of the divided storyboard to include the divided storyboard in the target matching scene.
[0024] In one embodiment, the first scene is earlier than the second scene, and if the target matching scene is the first scene, the tail scene boundary frame of the divided storyboard is set to be the tail scene boundary frame of the first scene, wherein the tail scene boundary frame of the first scene is the target scene boundary frame; or
[0025] The first scene is earlier than the second scene. If the target matching scene is the second scene, the head scene boundary frame of the divided scene is set to the head scene boundary frame of the second scene, wherein the head scene boundary frame of the second scene is the target scene boundary frame.
[0026] In one embodiment, the generating of the hierarchical marking information of the target image-caption combination based on the segment identifier of the target unit segment includes at least one of the following situations:
[0027] If the unit segment is a storyboard, the storyboard identifier of the target storyboard is set to the frame storyboard identifier of the target image frame as the hierarchical marking information; wherein the target storyboard is the unit segment to which the target image frame included in the target image subtitle combination belongs;
[0028] If the unit segment is a scene, the scene identifier of the target scene is set as the frame scene identifier of the target image frame as the hierarchical marking information; wherein the target scene is the unit segment to which the target image frame included in the target image subtitle combination belongs;
[0029] If the unit segment includes a scene and a storyboard, the storyboard scene marking information of the target image subtitle combination is generated according to the frame storyboard identifier and the frame scene identifier as the hierarchical marking information.
[0030] In one embodiment, the method for generating the image-caption combination includes:
[0031] Based on the subtitle file and the image sequence included in the video file, alignment processing is performed on the time sequence to obtain time-aligned image frames and subtitle texts;
[0032] The image-caption combination is generated based on the time-aligned image frames and caption texts.
[0033] In one embodiment, the subtitle file corresponds to a plurality of subtitle texts; the subtitle file and the image sequence included in the video file are aligned in time sequence to obtain time-aligned image frames and subtitle texts, including:
[0034] In the image sequence, determining a matching image frame located in a time period corresponding to each subtitle text;
[0035] The time-aligned image frame and subtitle text are determined based on each subtitle text and the matching image frame.
[0036] In one embodiment, the determining the time-aligned image frame and subtitle text based on each subtitle text and the matching image frame includes:
[0037] If there are multiple matching image frames, screening and / or merging the matching image frames to obtain a representative image frame matching each subtitle text;
[0038] The time-aligned image frame and subtitle text are determined based on each subtitle text and the representative image frame.
[0039] In one embodiment, the method further comprises:
[0040] In the process of segmenting the video file, for a first image frame sequence that does not match a subtitle text, if the number of frames of the first image frame sequence meets a preset frame discarding condition, the first image frame sequence is discarded.
[0041] In one embodiment, segmenting the video file to obtain a video clip including at least one unit segment includes:
[0042] Determine a preset range of image frame quantity and image subtitle combination quantity range;
[0043] The video file is segmented according to the image frame quantity range and the image subtitle combination quantity range to obtain the video segments.
[0044] The present disclosure provides a method for processing a video file, the method comprising:
[0045] Acquire an image-subtitle combination corresponding to the video file; wherein the video file is divided into a plurality of unit segments; the image-subtitle combination is obtained based on an image frame in the video file and a subtitle text time-aligned with the image frame;
[0046] The video file is segmented according to a preset video segmentation condition to obtain a video clip including at least one unit segment; wherein the video clip corresponds to a target image-subtitle combination; the unit segment to which the target image frame included in the target image-subtitle combination belongs is the target unit segment; the target image-subtitle combination corresponds to hierarchical marking information, and the hierarchical marking information is used to characterize the hierarchical structure of the target image-subtitle combination in the video file; the hierarchical marking information is generated based on the segment identifier of the target unit segment.
[0047] The present specification provides a method for training a visual narrative reasoning model, the method comprising:
[0048] Obtain a data set sample constructed based on the method described in any one of the above implementation modes;
[0049] The initial visual narrative reasoning model is trained using the data set samples until the model training stop condition is met, thereby obtaining a target visual narrative reasoning model.
[0050] In one embodiment, the visual narrative reasoning model is any one of a video model set; the video model set includes an action prediction model, a video question-answering model, and a video-based dialogue generation model.
[0051] The present specification provides a data set sample construction device, the device comprising:
[0052] An image-caption combination acquisition module is used to acquire an image-caption combination corresponding to a video file; wherein the video file is divided into a plurality of unit segments; the image-caption combination is obtained based on an image frame in the video file and / or a caption text aligned with the image frame in time;
[0053] A video segmentation module is used to segment the video file to obtain a video segment including at least one unit segment; wherein the video segment corresponds to a target image-caption combination; the unit segment to which the target image frame included in the target image-caption combination belongs is the target unit segment;
[0054] A marking information generating module, used to generate hierarchical marking information of the target image-subtitle combination based on the segment identifier of the target unit segment; wherein the hierarchical marking information is used to represent the hierarchical structure of the target image-subtitle combination in the video file;
[0055] The data set sample construction module is used to construct the data set sample based on the hierarchical labeling information of the video clip and the target image subtitle combination.
[0056] The embodiment of the present specification provides a video file processing device, the device comprising:
[0057] An image-caption combination acquisition module is used to acquire an image-caption combination corresponding to the video file; wherein the video file is divided into a plurality of unit segments; the image-caption combination is obtained based on an image frame in the video file and a caption text aligned with the image frame in time;
[0058] A video segmentation module is used to segment the video file according to a preset video segmentation condition to obtain a video clip including at least one unit segment; wherein the video clip corresponds to a target image-subtitle combination; the unit segment to which the target image frame included in the target image-subtitle combination belongs is the target unit segment; the target image-subtitle combination corresponds to hierarchical marking information, and the hierarchical marking information is used to characterize the hierarchical structure of the target image-subtitle combination in the video file; the hierarchical marking information is generated based on the segmentation identifier of the target unit segment.
[0059] The present specification provides a visual narrative reasoning model training device, the device comprising:
[0060] A data set sample construction module, used to obtain a data set sample constructed based on the method described in any of the above implementation modes;
[0061] The visual narrative reasoning model training module is used to train the initial visual narrative reasoning model using the data set samples until the model training stop condition is met to obtain the target visual narrative reasoning model.
[0062] An embodiment of the present specification provides a computer device, which includes: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the steps of the method described in any of the above embodiments.
[0063] The embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.
[0064] An embodiment of the present specification provides a computer program product, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method described in any one of the above embodiments.
[0065] In the above-mentioned implementation of the specification, the image-caption combination corresponding to the video file is obtained, the video file is segmented to obtain a video segment including at least one unit segment, hierarchical marking information of the target image-caption combination is generated based on the segment identifier of the target unit segment, and a data set sample is constructed based on the hierarchical marking information of the video segment and the target image-caption combination. By generating the hierarchical marking information of the target image-caption combination, the annotation of the data set sample can be enriched, and the visual narrative structure of the video segment can be better determined through the constructed data sample set, so as to understand the overall content of the video segment. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 A schematic diagram of a process for constructing a data set sample provided in an embodiment of this specification;
[0067] Figure 2 A schematic diagram of a process for determining a scene boundary frame provided in an embodiment of this specification;
[0068] Figure 3 A schematic diagram of a process for determining a target matching scenario provided in an embodiment of this specification;
[0069] Figure 4 A schematic diagram of a process for generating an image subtitle combination provided in an embodiment of this specification;
[0070] Figure 5 A schematic diagram of a process for matching image frames and subtitle texts provided in an embodiment of this specification;
[0071] Figure 6 A schematic diagram of a process for matching image frames and subtitle texts provided in an embodiment of this specification;
[0072] Figure 7 A schematic diagram of a process for obtaining a video clip provided in an embodiment of this specification;
[0073] Figure 8 A schematic diagram of a process for obtaining a video clip provided in an embodiment of this specification;
[0074] Fig. 9 A schematic diagram of a process for training a visual narrative reasoning model provided in an embodiment of this specification;
[0075] Fig.10 A schematic diagram of a data set sample construction device provided in an embodiment of this specification;
[0076] Fig.11 A schematic diagram of a video file processing device provided in an embodiment of this specification;
[0077] Fig.12A schematic diagram of a visual narrative reasoning model training device provided in an embodiment of this specification;
[0078] Fig.13 An internal structural diagram of a computer device provided for an embodiment of this specification. DETAILED DESCRIPTION
[0079] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0080] Although video understanding technology has made some progress in recent years, researchers in recent years have regarded video understanding tasks as visual recognition and classification tasks. After analysis, the inventors found that for long video files (such as movies and TV shows), the computational cost is huge and growing, which will also limit the development of this type of video understanding. Furthermore, after analysis, the inventors found that two key factors are ignored in the relevant technology: overall video understanding and the visual narrative structure of the video.
[0081] Different from the visual recognition and classification that are mainly concerned in the related art, the inventor proposes a method for understanding videos in a holistic and abstract way to make up for the deviation from the natural environment of understanding videos in a holistic and abstract way in the related art, that is, to regard the video understanding task as a multi-frame image understanding problem for modeling internal frames and inter-frames. In the implementation mode of this specification, it is proposed that in addition to individual visual surfaces and subtitle vocabulary, the ability to perceive and understand multimodal image sequences is closely related to the basic narrative structure, which is a hierarchical understanding that guides the presentation of events and individual concepts.
[0082] In order to solve the above problems, the task of creating a new dataset is proposed. By rearranging the scenes, storyboards, and frames of video clips in the presence of video dialogue information, a dataset is constructed on this basis. The video understanding model is trained with this dataset to improve the temporal feature learning and structural learning of the video understanding model.
[0083] Based on this, the embodiment of this specification provides a method for constructing a data set sample, obtaining an image-caption combination corresponding to a video file, segmenting the video file to obtain a video segment including at least one unit segment, generating hierarchical marking information of a target image-caption combination based on the segment identifier of the target unit segment, and constructing a data set sample based on the hierarchical marking information of the video segment and the target image-caption combination. By generating the hierarchical marking information of the target image-caption combination, the annotation of the data set sample can be enriched, and the visual narrative structure of the video segment can be better determined through the constructed data sample set, thereby understanding the overall content of the video segment.
[0084] This specification provides a method for constructing a data set sample. Figure 1 , the method may include the following steps:
[0085] S110: Obtain an image-subtitle combination corresponding to the video file.
[0086] S120: Segment the video file to obtain video clips including at least one unit segment.
[0087] Among them, the video file is divided into several unit segments. The unit segment can be a division unit of the video file, and the unit segment can be a video that can be used to describe a simple or short plot. The unit segment can be a scene, a storyboard, or a grid of an animation video. The image subtitle combination is obtained based on the image frame in the video file and / or the subtitle text aligned with the image frame time. The image subtitle combination can include the image frame in the video file, or it can include the image frame in the video file and the subtitle text aligned with its time. The image subtitle combination can also be called an image subtitle pair. The subtitle text can be text added to the video file later, such as the title, cast, lyrics, dialogue, description, character introduction, place name and year of the film. The type of the video file can be any one of animation, short video, 3D movie, etc. The image frame can be a single image picture that is the smallest unit in the image animation, and a frame can be a still picture.
[0088] The video clip corresponds to a target image subtitle combination, and the unit segment to which the target image frame included in the target image subtitle combination belongs is the target unit segment.
[0089] Specifically, a video file includes an image frame sequence and a subtitle file. The image frame sequence and the subtitle file are time-aligned. Some image frames can be matched to subtitle texts that are time-aligned, while some image frames are not matched to subtitle texts that are time-aligned. Thus, an image-subtitle combination corresponding to the video file is obtained, and the image-subtitle combination includes the image frames in the video file and / or the subtitle texts that are time-aligned with the image frames.
[0090] The production process of a video file corresponds to the division of unit segments, so a video file can be divided into several unit segments. In order to improve the completeness of the story described by the dataset samples, such as the complete expression of the story plot, event plot, character actions, etc., when segmenting the video, a complete unit segment is divided into a video segment as much as possible, that is, a video segment includes at least one complete unit segment, so that the video clip can describe a short plot.
[0091] S130: Generate hierarchical marking information of a target image-caption combination based on the segment identifier of the target unit segment.
[0092] S140, constructing a data set sample based on the hierarchical labeling information of the video clip and the target image subtitle combination.
[0093] The hierarchical marking information is used to characterize the hierarchical structure of the target image subtitle combination in the video file. The hierarchical structure can be determined based on the unit segmentation divided in the production process of the video file. For example, if the video file is divided into multiple scenes during production, the hierarchical structure is scene-image frame (Scene+rame). For example, if the video file is divided into multiple shots during production, the hierarchical structure is shot-image frame (Shot+Frame). For example, if the video file is divided into multiple scenes during production, and each scene is divided into multiple shots, the hierarchical structure is scene-shot-image frame (Scene+Shot+Frame).
[0094] Specifically, after segmenting the video file, a video clip including at least one unit segment can be obtained. The video file corresponds to an image-subtitle combination, and the image-subtitle combination aligned with the time of the video segment can be regarded as a number of target image-subtitle combinations corresponding to the video clip. Each target image-subtitle combination includes an image frame, and the unit segment to which the image frame belongs is recorded as a target unit segment. Exemplarily, the target unit segment corresponding to the image frame can be a target storyboard, the target unit segment corresponding to the image frame can be a target scene, and the target unit segment corresponding to the image frame can be a target storyboard and a target scene.
[0095] In this embodiment, a segmentation identifier is provided for the unit segment. If an image frame belongs to a unit segment, the attribute information of the image frame may include the segmentation identifier of the unit segment to which the image frame belongs. The segmentation identifier can be used to characterize the hierarchical structure of the image frame in the video file, that is, which unit segment it belongs to. Furthermore, in order to improve the accuracy of understanding the video content, this embodiment combines data of two modes, namely, image frames and their corresponding subtitle texts. Therefore, it is necessary to set the segmentation identifier of the target unit segment to which the target image-subtitle combination belongs as the attribute information of the target image-subtitle combination. Specifically, the hierarchical marking information of the target image-subtitle combination is generated based on the segmentation identifier of the target unit segment to describe the hierarchical structure of the target image-subtitle combination in the video file, that is, which unit segment the target image-subtitle combination belongs to, so as to facilitate the organization and understanding of the video content.
[0096] In this embodiment, a video clip corresponds to a plurality of target image subtitle combinations, and each target image subtitle combination has its own hierarchical marking information. The video clip is annotated based on the hierarchical marking information of each target image subtitle combination corresponding to the video clip, and the hierarchical structure of the video clip in the video file is obtained. A complete and concise plot is described through the video clip, which provides a basis for understanding the content of the video file from a holistic perspective.
[0097] In the above implementation, the image-caption combination corresponding to the video file is obtained, the video file is segmented to obtain a video segment including at least one unit segment, hierarchical marking information of the target image-caption combination is generated based on the segment identifier of the target unit segment, and a data set sample is constructed based on the hierarchical marking information of the video segment and the target image-caption combination. By generating the hierarchical marking information of the target image-caption combination, the annotation of the data set sample can be enriched, and the visual narrative structure of the video segment can be better determined through the constructed data sample set, so as to understand the overall content of the video segment.
[0098] It should be noted that most video files have two inherent hierarchical structures: shots and scenes. A shot can be shot using a fixed camera angle, and a shot is usually a series of temporally consecutive image frames, representing several actions. A scene can be a series of shots sharing the same environmental background, usually depicting an event or a short story. Obtaining the hierarchical structure of a video file is crucial to understanding the video file.
[0099] In some implementations, the unit segment is a scene, and the unit segment to which the target image frame included in the target image-caption combination belongs is the target scene. Generating hierarchical tag information of the target image-caption combination based on the segment identifier of the target unit segment includes: generating hierarchical tag information of the target image-caption combination based on the scene identifier of the target scene.
[0100] In some cases, if the hierarchical structure of a video file is scene-image frame, hierarchical marking information in the form of scene-image frame can be generated for the image-subtitle combination. Among them, a scene corresponds to a scene identifier. If an image frame belongs to a scene, the attribute information of the image frame may include the scene identifier to which the image frame belongs. The scene identifier can be used to characterize the hierarchical structure of the image frame in the video file, that is, which scene it belongs to. This embodiment combines data of two modes, image frame and its corresponding subtitle text, and therefore, it is necessary to set the scene identifier to which the target image-subtitle combination belongs as the attribute information of the target image-subtitle combination.
[0101] Specifically, hierarchical tag information of the target image-subtitle combination is generated based on the scene identification of the target scene to describe the hierarchical structure of the target image-subtitle combination in the video file, that is, to which scene the target image-subtitle combination belongs, so as to facilitate the organization and understanding of the video content.
[0102] For example, Video Scene Segmentation (VSS) can segment a long video file into several independent sub-videos based on the shot dimension according to the different scene contents. The segmentation process is also called Scene Boundary Detection.
[0103] In the above implementation, hierarchical marking information of the target image-caption combination is generated based on the scene identification of the target scene, so as to hierarchically process the video file according to the narrative structure of frames and scenes, thereby retaining the narrative structure of the video file.
[0104] In some implementations, the unit segment is a storyboard, and the unit segment to which the target image frame included in the target image-subtitle combination belongs is the target storyboard. Generating hierarchical marking information of the target image-subtitle combination based on the segmentation identifier of the target unit segment includes: generating hierarchical marking information of the target image-subtitle combination based on the storyboard identifier of the target storyboard.
[0105] In some cases, if the hierarchical structure of a video file is storyboard-image frame, hierarchical marking information in the form of storyboard-image frame can be generated for the image-subtitle combination. Among them, a storyboard corresponds to a storyboard identifier. If an image frame belongs to a storyboard, the attribute information of the image frame may include the storyboard identifier of the storyboard to which the image frame belongs. The storyboard identifier can be used to characterize the hierarchical structure of the image frame in the video file, that is, which storyboard it belongs to. This embodiment combines data of two modes, image frame and its corresponding subtitle text, and therefore, it is necessary to set the storyboard identifier of the storyboard to which the target image-subtitle combination belongs as the attribute information of the target image-subtitle combination.
[0106] Specifically, hierarchical tag information of the target image-subtitle combination is generated based on the storyboard identifier of the target storyboard to describe the hierarchical structure of the target image-subtitle combination in the video file, that is, which storyboard the target image-subtitle combination belongs to, so as to facilitate the organization and understanding of the video content.
[0107] Exemplarily, the corresponding storyboards of the video file can be obtained by processing the video file using the TransNetV2 shot boundary detection method.
[0108] It should be noted that after the video file is storyboarded, the video file can be divided into several storyboards. There can be multiple image frames in each storyboard, and 1 to 3 image frames can be screened out from the multiple image frames in each storyboard. The screening principle can be that when any image frame does not have a corresponding subtitle text, the image frames before and after the image frame can be determined. If the image frames before and after the image frame have corresponding subtitle texts, it can be considered that the image frame is not very helpful for understanding the content of the video file, so the image frame can be discarded. When any image frame does not have a corresponding subtitle text, the image frames before and after the image frame can be determined. If the image frames before and after the image frame also do not have corresponding subtitle texts and the content of the image frame can represent a part of the plot content, it can be considered that the image frame is very helpful for understanding the content of the video file, so it can be considered that the image frame needs to be retained. Among them, determining whether the image frames before and after the image frame have corresponding subtitle texts can be determined by writing a script.
[0109] In the above implementation, hierarchical marking information of the target image subtitle combination is generated based on the storyboard identifier of the target storyboard, so as to hierarchically process the video file according to the narrative structure of frames and storyboards, thereby retaining the narrative structure of the video file.
[0110] In some embodiments, the unit segment includes a scene and a storyboard, the storyboard to which the target image frame included in the target image-caption combination belongs is the target storyboard, and the scene to which the target storyboard belongs is the target scene. Generating hierarchical marking information of the target image-caption combination based on the segmentation identifier of the target unit segment may include: generating hierarchical marking information of the target image-caption combination based on the storyboard identifier of the target storyboard and the scene identifier of the target scene.
[0111] In some cases, the hierarchical structure of the video file is scene-storyboard-image frame, and hierarchical marking information in the form of scene-storyboard-image frame can be generated for the image-subtitle combination. Among them, the scene corresponds to a scene identifier, and the storyboard corresponds to a storyboard identifier. If an image frame belongs to a storyboard, and the storyboard belongs to a scene, the attribute information of the image frame may include the storyboard identifier of the storyboard to which the image frame belongs, and the scene identifier to which the image frame belongs. The storyboard identifier and the scene identifier can be used to characterize the hierarchical structure of the image frame in the video file, that is, which storyboard and which scene it belongs to. This embodiment uses data of two modes, image frames and their corresponding subtitle texts, to combine, therefore, it is necessary to set the storyboard identifier of the storyboard to which the target image-subtitle combination belongs and the scene identifier to which it belongs as the attribute information of the target image-subtitle combination.
[0112] Specifically, hierarchical tagging information of the target image-subtitle combination is generated based on the storyboard identifier of the target storyboard and the scene identifier of the target scene to describe the hierarchical structure of the target image-subtitle combination in the video file, that is, which storyboard and scene the target image-subtitle combination belongs to, so as to facilitate the organization and understanding of the video content.
[0113] In the above implementation, hierarchical tag information of the target image subtitle combination is generated based on the shot identification of the target shot and the scene identification of the target scene, so as to hierarchically process the video file according to the narrative structure of frame, shot, and scene, and retain the narrative structure of the video file. When a video understanding model is trained using a data set with the hierarchical tag information of scene-shot-frame, the video understanding model can learn to understand the content of the video content in an overall manner and have the ability to understand the hierarchical structure of the video file.
[0114] In some embodiments, see Figure 2 , the video file includes a first scene, and may include the following steps:
[0115] S210: Determine an initial scene boundary frame of the first scene and a storyboard boundary frame of the storyboard.
[0116] Specifically, in some embodiments, the video file can be divided into scenes by a machine learning model algorithm to obtain an initial scene boundary frame of the first scene. The video file can be divided into storyboards by a machine learning model algorithm to obtain storyboard boundary frames of the storyboards. In some other embodiments, the initial scene boundary frame of the first scene and the storyboard boundary frames of the storyboards can be directly obtained based on a data set with boundary annotation information in the related art (such as a Movienet data set).
[0117] S220: If the initial scene boundary frame of the first scene is not a storyboard boundary frame of any storyboard, determine any storyboard as a divided storyboard.
[0118] S230, performing boundary alignment processing based on the storyboard boundary frames of the divided storyboards and the initial scene boundary frame of the first scene to obtain a target scene boundary frame of the first scene.
[0119] The target scene boundary frame is used as a reference for segmenting the video file.
[0120] In some cases, whether it is the initial scene boundary frame determined by scene division through the machine learning model algorithm or the initial scene boundary frame obtained from the data set in the related art, the initial scene boundary frame should be the storyboard boundary frame of a certain storyboard. If not, the initial scene boundary frame passes through the certain storyboard. Such division will inevitably destroy the integrity of the complete and brief plot described by the certain storyboard, which is inevitably unreasonable. Therefore, in this embodiment, detection is performed for this situation and a corresponding boundary alignment processing method is proposed to reduce the damage to the integrity of the storyboard.
[0121] Specifically, each storyboard corresponds to two storyboard boundary frames, and it is detected whether the initial scene boundary frame of the first scene is the storyboard boundary frame of any storyboard. If not, it indicates that the initial scene boundary frame passes through or divides any storyboard, so any storyboard can be determined as a divided storyboard, and the initial scene boundary frame also needs to be aligned. Therefore, the boundary alignment process is performed according to the storyboard boundary frame of the divided storyboard and the initial scene boundary frame of the first scene to obtain the target scene boundary frame of the first scene.
[0122] Exemplarily, any one of the two storyboard boundary frames of the divided storyboard can be set as the initial scene boundary frame, so that the initial scene boundary frame will no longer pass through or divide the divided storyboard. Exemplarily, the storyboard boundary frame close to the initial scene boundary frame can be determined among the two storyboard boundary frames of the divided storyboard, and the storyboard boundary frame close to the initial scene boundary frame can be set as the initial scene boundary frame, so that the initial scene boundary frame will no longer pass through or divide the divided storyboard.
[0123] In this implementation, the video file can be divided into boundaries by a machine learning model algorithm to determine the initial scene boundary frame of the scene and the storyboard boundary frame of the storyboard. It can also be directly obtained from an open source data set, which has a frame-to-shot boundary and a frame-to-scene boundary. The open source data set can be MovieNet.
[0124] In the above implementation, the initial scene boundary frame of the first scene and the storyboard boundary frame of the storyboard are determined. If the initial scene boundary frame of the first scene is not the storyboard boundary frame of any storyboard, any storyboard is determined to be a divided storyboard, and boundary alignment processing is performed based on the storyboard boundary frame of the divided storyboard and the initial scene boundary frame of the first scene to obtain the target scene boundary frame of the first scene. By determining the boundary frame of the scene, the video file can be better segmented subsequently, so that the integrity of the video narrative can be guaranteed when the video file is segmented subsequently.
[0125] In some embodiments, see Figure 3 The video file includes a second scene adjacent to the first scene. Performing boundary alignment processing based on the storyboard boundary frames of the divided storyboards and the initial scene boundary frame of the first scene to obtain the target scene boundary frame of the first scene may include the following steps:
[0126] S310: Determine a first matching situation between the divided storyboard and the first scene, and a second matching situation between the divided storyboard and the second scene.
[0127] S320: Determine a target matching scene that matches the divided storyboard in the first scene and the second scene according to the first matching situation and the second matching situation.
[0128] S330. Setting a target scene boundary frame according to the storyboard boundary frame of the divided storyboard to include the divided storyboard in the target matching scene.
[0129] Among them, the matching situation can be determined based on the correlation or similarity between the storyboard content and the scene content. A preset matching threshold is set for the matching situation. Exceeding the matching threshold can indicate that there is correlation or similarity between the storyboard content and the scene content, and not exceeding the matching threshold can indicate that there is no correlation or dissimilarity between the storyboard content and the scene content.
[0130] In some cases, the first scene has an adjacent second scene. As mentioned above, the initial scene boundary frame of the first scene needs to be adjusted. In order to improve the accuracy of the data set samples, it is necessary to determine whether the divided storyboard belongs to the first scene or the second scene. If it is determined that the divided storyboard belongs to the first scene, then each image frame included in the divided storyboard is divided into the first scene. If it is determined that the divided storyboard belongs to the second scene, then each image frame included in the divided storyboard is divided into the second scene.
[0131] Specifically, in some embodiments, image similarity data between the image frames included in the divided storyboard and the image frames included in the first scene are calculated to obtain a first matching situation between the divided storyboard and the first scene. Image similarity data between the image frames included in the divided storyboard and the image frames included in the second scene are calculated to obtain a second matching situation between the divided storyboard and the second scene. In some other embodiments, graphic and text similarity data between the image-subtitle combination included in the divided storyboard and the image-subtitle combination included in the first scene are calculated to obtain a first matching situation between the divided storyboard and the first scene. Graphic and text similarity data between the image-subtitle combination included in the divided storyboard and the image-subtitle combination included in the second scene are calculated to obtain a second matching situation between the divided storyboard and the second scene.
[0132] The first matching situation and the second matching situation are compared, and a target matching scene matching the divided storyboard is determined in the first scene and the second scene according to the comparison result of the two. If the second matching situation is higher than the first matching situation, the second scene is determined to be the target matching scene. If the first matching situation is higher than the second matching situation, the first scene is determined to be the target matching scene. Further, since the divided storyboard is similar to the target matching scene or has a high correlation, it is necessary to set the target scene boundary frame according to the storyboard boundary frame of the divided storyboard so as to include the divided storyboard in the target matching scene.
[0133] Exemplarily, when the initial scene boundary frame of the first scene is not a frame boundary frame of any frame, the initial scene boundary frame of the second scene is not a frame boundary frame of any frame, and the initial scene boundary frame of the first scene and the initial scene boundary frame of the second scene are in the same frame, the frame can be determined as a divided frame. The first matching situation between the divided frame and the first scene can be determined according to the correlation between the content of the divided frame and the content of the first scene, and the second matching situation between the divided frame and the second scene can be determined according to the correlation between the content of the divided frame and the content of the second scene. The first matching situation and the second matching situation can be compared, and when the first matching situation is greater than the second matching situation and the first matching situation exceeds a preset matching threshold, the first scene can be determined as a target matching scene that matches the divided frame, and when the second matching situation is greater than the first matching situation and the second matching situation exceeds a preset matching threshold, the second scene can be determined as a target matching scene that matches the divided frame. Then, the target scene boundary frame can be set according to the frame boundary frame of the divided frame, and after the target scene boundary frame is set, the divided frame can be included in the target matching scene.
[0134] It should be noted that when both the first matching situation and the second matching situation fail to reach the expected matching threshold, the non-storyboard boundary frame in the divided storyboard can be discarded, and the image frame after the tail storyboard boundary frame of the divided storyboard can be used as the head scene boundary frame of the second scene, and the image frame before the head storyboard boundary frame of the divided storyboard can be used as the tail scene boundary frame of the first scene.
[0135] In the above implementation, a first matching situation between the divided storyboard and the first scene, and a second matching situation between the divided storyboard and the second scene are determined, and a target matching scene matching the divided storyboard is determined in the first scene and the second scene based on the first matching situation and the second matching situation, and a target scene boundary frame is set based on the storyboard boundary frame of the divided storyboard to include the divided storyboard in the target matching scene, thereby ensuring that the scene and the storyboard match to the greatest extent, improving the probability of the plot and events fully described by the video clip, and improving the reliability of the data set sample.
[0136] In some embodiments, the first scene is earlier than the second scene. If the target matching scene is the first scene, the tail shot boundary frame of the divided shot is set to the tail scene boundary frame of the first scene, wherein the tail scene boundary frame of the first scene is the target scene boundary frame; or, the first scene is earlier than the second scene. If the target matching scene is the second scene, the head shot boundary frame of the divided shot is set to the head scene boundary frame of the second scene, wherein the head scene boundary frame of the second scene is the target scene boundary frame.
[0137] Specifically, the first scene is earlier than the second scene. When the target matching scene that matches the divided storyboard is the first scene, the tail storyboard boundary frame of the divided storyboard can be set as the tail scene boundary frame of the first scene, and the next image frame of the tail storyboard boundary frame of the divided storyboard can be used as the head scene boundary frame of the second scene, so that the divided storyboard is included in the first scene.
[0138] The first scene is earlier than the second scene. When the target matching scene that matches the divided storyboard is the second scene, the head storyboard boundary frame of the divided storyboard can be set as the head scene boundary frame of the second scene, and the previous image frame of the head storyboard boundary frame of the divided storyboard can be used as the tail scene boundary frame of the first scene to include the divided storyboard in the second scene.
[0139] In the above implementation, the first scene is earlier than the second scene. If the target matching scene is the first scene, the tail shot boundary frame of the divided shot is set to the tail scene boundary frame of the first scene; or the first scene is earlier than the second scene. If the target matching scene is the second scene, the head shot boundary frame of the divided shot is set to the head scene boundary frame of the second scene, so as to ensure the continuity and rationality of the content between the scenes and the shots as much as possible, improve the probability of the plot and events fully described by the video clip, and improve the reliability of the data set samples.
[0140] In some embodiments, the hierarchical marking information of the target image-subtitle combination is generated based on the segmentation identifier of the target unit segmentation, including at least one of the following situations: if the unit segmentation is a storyboard, the storyboard identifier of the target storyboard is set to the frame storyboard identifier of the target image frame as the hierarchical marking information. Alternatively, if the unit segmentation is a scene, the scene identifier of the target scene is set to the frame scene identifier of the target image frame as the hierarchical marking information. Alternatively, if the unit segmentation includes a scene and a storyboard, the storyboard scene marking information of the target image-subtitle combination is generated according to the frame storyboard identifier and the frame scene identifier as the hierarchical marking information.
[0141] Among them, the target storyboard is a unit segment to which the target image frame included in the target image-caption combination belongs, and the target scene is a unit segment to which the target image frame included in the target image-caption combination belongs.
[0142] Specifically, when the unit segment is a storyboard, by determining the storyboard to which the target image frame belongs, the storyboard identifier of the storyboard can be set as the frame storyboard identifier of the target image frame, and the frame storyboard identifier of the target image-subtitle combination can be generated based on the frame storyboard identifier, and the frame storyboard identifier of the image-subtitle combination can be used as the hierarchical marking information of the target image frame.
[0143] When the unit segment is a scene, by determining the scene to which the target image frame belongs, the scene identifier of the scene can be set as the frame scene identifier of the target image frame. According to the frame scene identifier, the frame scene identifier of the target image-subtitle combination can be generated, and the frame scene identifier of the image-subtitle combination can be used as the hierarchical marking information of the target image frame.
[0144] When the unit segment includes a scene and a storyboard, by determining the storyboard to which the target image frame belongs, the storyboard identifier of the storyboard can be set as the frame storyboard identifier of the target image frame. By determining the storyboard to which the target image frame belongs, the scene to which the storyboard belongs can be determined, and the scene identifier of the scene can be set as the frame scene identifier of the target image frame. The storyboard scene marking information of the target image subtitle combination can be generated according to the frame storyboard identifier and the frame scene identifier, and the storyboard scene marking information can be used as the hierarchical marking information of the target image frame.
[0145] In the above implementation, after determining the frame storyboard identifier to which the target image frame belongs or determining the frame scene identifier to which the target image frame belongs or determining the frame storyboard identifier to which the target image frame belongs, the frame scene identifier to which the target image frame belongs can be determined according to the frame scene identifier to which the frame storyboard identifier belongs, and the video file can be processed hierarchically according to the narrative structure of frames, storyboards, and scenes, thereby retaining the narrative structure of the video file.
[0146] In some embodiments, see Figure 4 , the generation method of the image caption combination may include the following steps:
[0147] S410: Based on the subtitle file and the image sequence included in the video file, an alignment process is performed on the time sequence to obtain time-aligned image frames and subtitle texts.
[0148] S420: Generate an image-caption combination based on the time-aligned image frames and caption texts.
[0149] The image sequence may be a sequence composed of continuous image frames.
[0150] Specifically, a video file may include a subtitle file and an image sequence. The subtitle file can be aligned with the image frames contained in the image sequence in a time sequence in combination with the frame rate and the timestamp of the subtitle file to obtain time-aligned image frames and subtitle texts. Then, an image-subtitle combination can be generated based on the time-aligned image frames and subtitle texts.
[0151] In some embodiments, since the hierarchical structure of multimodal video files needs to be refined and reordered, frame-level annotations are required. Exemplarily, the video file can be a movie, and the video file can use the movie in the existing large-scale movie dataset pool MovieNet. The video files in the original pool can be screened by the frame images, frame-to-shot boundaries, frame-to-scene boundaries, subtitle files corresponding to the video files, actor IDs and introductions present in the video files, and video files with rich tags and complete semantic structures can be obtained. Among them, movies with less than 500 dialogues will be discarded, because movies with less than 500 dialogues have less text content, so the effect of the movie for visual narrative reasoning is not obvious, so movies with less than 500 dialogues can be discarded. Movies with less than 500 dialogues can be considered to be documentaries or thrillers.
[0152] The image frames extracted from the video files with rich tags and complete semantic structures can be RGB images extracted from the 240P video files using screenshots and cropped to remove the black borders. In terms of subtitles, an English subtitle file aligned with the video file (such as a movie version) can be selected. For subtitle text, symbols that are not in Unicode can be discarded. If the subtitle text is not in English, it can be translated into English text, and onomatopoeic subtitles and narrations representing ambient sounds can be filtered out. The annotation of each frame of the movie can include the name of the movie corresponding to each frame, the number of shots of the movie for each frame, the number of scenes of the movie for each frame, and the plot told by each frame.
[0153] In the above implementation, based on the subtitle file and the image sequence included in the video file, alignment processing is performed on the time sequence to obtain time-aligned image frames and subtitle texts. In the image-subtitle combination generated based on the time-aligned image frames and subtitle texts, the image frames and subtitle texts are aligned with each other in time, which lays an accurate time foundation for the structure of the visual narrative structure of the subsequent data set samples, which is conducive to improving the quality of the data set samples. Furthermore, when using the data set samples to train the video understanding model, the accuracy of the video understanding model can be improved.
[0154] In some implementations, a subtitle file corresponds to multiple subtitle texts. Figure 5 Based on the subtitle file and the image sequence included in the video file, alignment processing is performed on the time sequence to obtain the time-aligned image frames and subtitle texts, which may include the following steps:
[0155] S510: In an image sequence, determine a matching image frame located in a time period corresponding to each subtitle text.
[0156] S520: Determine time-aligned image frames and subtitle texts based on each subtitle text and the matching image frame.
[0157] In some cases, the subtitle text corresponds to a timestamp, which is usually a sequence of characters that uniquely identifies a certain moment in time. Each image frame corresponds to a timestamp, and each subtitle text corresponds to a timestamp, so the image frame can be matched with the subtitle text through the timestamp. It should be noted that the image frame corresponds to a frame number, and the frame number can be used to identify the image frame from the header image frame of the original video file. For example, if the frame number is 3, it can be said that the image frame is the third image frame from the header image frame of the original video file. Among them, the frame number of the header image frame of the original video file can be 1. The timestamp is usually a relative timestamp, that is, relative to the starting time point of the video file. If you need to convert the timestamp into an absolute timestamp, you need to know the starting time point of the video file and add the corresponding offset. The timestamp unit can be seconds or milliseconds, so the subtitle text timestamp and the image frame timestamp may be inclusive. For example, if the subtitle text timestamp is 00:23:15:00, the image frame timestamp can be 00:23:15:05, which means that the subtitle text contains image frames between 00:23:15 and 00:23:16.
[0158] Specifically, according to the frame rate and frame sequence number, the timestamp corresponding to each image frame in the image sequence can be determined, and according to the timestamp corresponding to the subtitle text and the timestamp corresponding to the image frame, the matching image frame located within the timestamp corresponding to each subtitle text can be determined, and then the time-aligned image frame and subtitle text can be determined based on the subtitle text and the matching image frame.
[0159] Exemplarily, the frame rate of a video file may be 25fps, the timestamp of the first image frame with frame number 1 may be 00:00:00:00, the timestamp of the second image frame with frame number 2 may be 00:00:00:40 (1 / 25), and the timestamp of the third image frame with frame number 3 may be 00:00:00:80 (2 / 25).
[0160] In the above implementation, in the image sequence, the matching image frames located in the time period corresponding to each subtitle text are determined, and the time-aligned image frames and subtitle texts are determined based on each subtitle text and the matching image frames. This can lay an accurate time foundation for the structure of the visual narrative structure of the subsequent data set samples, and can improve the quality of the data set samples. Furthermore, when the data set samples are used to train the video understanding model, the accuracy of the video understanding model can be improved.
[0161] In some embodiments, see Figure 6 , determining the time-aligned image frames and subtitle texts based on each subtitle text and the matching image frame may include the following steps:
[0162] S610: If there are multiple matching image frames, screen and / or merge the matching image frames to obtain a representative image frame matching each subtitle text.
[0163] S620: Determine time-aligned image frames and subtitle texts based on each subtitle text and the representative image frame.
[0164] The representative image frame may be an image frame that best fits the content of the subtitle text among multiple image frames matching the subtitle text.
[0165] In some cases, the content expressed by the subtitle text is relatively rich, so the subtitle text stays for a longer time and corresponds to more image frames. Therefore, the number of image frames matching the subtitle text is multiple.
[0166] Specifically, when there are multiple image frames matching the subtitle text, when there are large differences between the image contents of the multiple image frames, the image frame content and the subtitle text content can be screened and processed according to the matching degree between the image frame content and the subtitle text content to obtain the image frame that best matches the subtitle text content and use it as a representative image frame matching the subtitle text. When there are no large differences between the image contents of the multiple image frames, the middle image frame of the multiple image frames can be used as a representative image frame matching the subtitle text. Then, the remaining image frames in the multiple image frames matching the subtitle text, that is, image frames that are not representative image frames, can be discarded. Then, the subtitle text and the representative image frames can be aligned in time to determine the time-aligned image frames and subtitle text. It should be noted that if there are multiple matching image frames, multiple matching image frames can also be fused to obtain a representative image frame matching each subtitle text.
[0167] In some embodiments, there may be multiple image frames with similar contents, wherein the number of image frames with corresponding subtitle texts is greater than 1. Therefore, the middle image frame of the multiple image frames may be used as a representative image frame, and the subtitle texts in the multiple image frames may be aligned with the time of the representative image frame, so that the subtitle text of the representative image frame contains the subtitle text content of the multiple image frames.
[0168] It should be noted that after the representative image frame is selected, if there are special image frames that play a role in visual narrative among the remaining image frames except the representative image frame, they can be retained.
[0169] In the above implementation, if there are multiple matching image frames, the matching image frames can be screened and processed together with the subtitle text to obtain representative image frames that match each subtitle text, and time-aligned image frames and subtitle texts can be determined based on each subtitle text and the representative image frames. Each image frame can be matched with its corresponding letter text to achieve the finest-grained alignment of image frames and subtitle texts in the time series, further limiting the maximum number of scene and storyboard transitions in the clip to reduce the computational cost of visual perception.
[0170] In some embodiments, the dataset sample construction method may further include: in the process of segmenting the video file, for a first image frame sequence that does not match the subtitle text, if the number of frames in the first image frame sequence meets a preset frame discarding condition, discarding the first image frame sequence.
[0171] The image frame sequence may be image frames arranged in sequence. The preset condition for discarding a frame may be that the number of image subtitle combinations corresponding to the image frame sequence does not meet a preset range of the number of image subtitle combinations.
[0172] In some cases, there may be consecutive image frames that do not match subtitle texts. In the process of visual narrative reasoning, consecutive image frames that do not match subtitle texts may not play a role in visual narrative reasoning, so the consecutive image frames that do not match subtitle texts can be discarded.
[0173] Specifically, by screening the matching image frames to obtain representative image frames that match each subtitle text, and then determining the time-aligned image frames and subtitle text based on each subtitle text and the representative image frames, the image frames of the video file can be screened to obtain the video file to be segmented. An identifier is set for each image frame in the video file to be segmented to distinguish whether the image frame in the video file to be segmented has a matched subtitle text. When the image frame in the video file to be segmented has a matched subtitle text, the identifier of the image frame can be set to identifier 1, and when the image frame in the video file to be segmented has no matched subtitle text, the identifier of the image frame can be set to identifier 0. Image frames are selected in sequence starting from the starting position of the video file to be segmented to obtain an image frame sequence. Exemplarily, screening starts from one image frame. When the identification of the image frame is 0, the image frames can be added to two image frames in sequence. If the identification of the image frame is still 0, the image frames can be added in sequence. When the number of selected image frames reaches 80% of the preset number of image frames and the identifications of the selected image frames are all 0, it can be considered that the image frame sequence does not contain text content that can be used for visual narrative reasoning, so the continuous image frame sequence with the identification of 0 can be discarded. In some cases, the next image frame of the discarded image frame sequence can be used as the starting position of the video file to be processed to continue to repeat the above operation until the entire video file is processed.
[0174] In the above implementation, in the process of segmenting the video file, for the first image frame sequence that does not match the subtitle text, if the number of frames of the first image frame sequence meets the preset frame discarding condition, the first image frame sequence is discarded. By discarding the first image frame sequence that does not match the subtitle text, image frames that do not involve visual narrative reasoning can be reduced, thereby reducing the consumption of computer resources.
[0175] In some embodiments, see Figure 7 , segmenting the video file to obtain a video clip including at least one unit segment may include the following steps:
[0176] S710: Determine a preset range of the number of image frames and a preset range of the number of image-caption combinations.
[0177] S720 , segment the video file according to the range of the number of image frames and the range of the number of image-subtitle combinations to obtain video segments.
[0178] Among them, the preset range of the number of image frames can be set according to the specific situation. For example, a general video clip can include 1 or 2 scenes, a scene can generally include 5 to 10 storyboards, and a storyboard can generally include 1 to 3 frames of images, so the preset range of the number of image frames can be 10 to 20 frames. The preset range of the number of image-subtitle combinations can be set according to the specific situation on the premise of ensuring that the semantics are not too sparse. For example, when the image-subtitle combination in the video clip is greater than 80%, it can be ensured that the semantics are not too sparse, so the range of the number of image-subtitle combinations can be 80% to 100%.
[0179] Specifically, the preset range of the number of image frames and the range of the number of image-subtitle combinations can be determined according to actual conditions. By screening the matching image frames, representative image frames matching each subtitle text are obtained, and then time-aligned image frames and subtitle texts are determined based on each subtitle text and the representative image frames. The image frames of the video file can be screened to obtain the video file to be segmented. An identifier is set for each image frame in the video file to be segmented to distinguish whether the image frame in the video file to be segmented has a matched subtitle text. When the image frame in the video file to be segmented has a matched subtitle text, the identifier of the image frame can be set to identifier 1, and when the image frame in the video file to be segmented does not have a matched subtitle text, the identifier of the image frame can be set to identifier 0. Select image frames in sequence from the starting position of the video file to be segmented, start screening from one image frame, when the number of selected image frames is within the preset image frame number range and the ratio between the image frame identified as identification 1 in the selected image frames and the selected image frame is within the preset image subtitle combination number range, continue to increase the number of image frames in sequence to two image frames, when the number of selected image frames is within the preset image frame number range and the ratio between the image frame identified as identification 1 in the selected image frames and the selected image frame is within the preset image subtitle combination number range, continue to increase the number of image frames in sequence until the number of selected image frames reaches the preset image frame number range and the ratio between the image frame identified as identification 1 in the selected image frames and the selected image frame is within the preset image subtitle combination number range, the video file to be segmented can be segmented at the position of the tail image frame of the selected image frame to obtain video segment A, and then the image frame after the image frame at the segmentation position can be used as the starting position of the video file to be processed to continue to repeat the above operation, the video file to be processed can be segmented to obtain video segments. Exemplarily, the screening process of the image frame can be completed by a greedy algorithm.
[0180] It should be noted that, in the process of selecting image frames, if the number of selected image frames is within the preset range of the number of image frames and the ratio between the image frames identified as identification 1 in the selected image frames and the selected image frames is within the preset range of the number of image subtitle combinations, if the tail image frame of the selected image frame sequence divides the scene of the video, it can be understood that the last scene corresponding to the selected image frame sequence does not contain all the image frames of the scene, so according to the specific situation, the tail image frame of the divided scene or the head frame of the scene can be used as the tail image frame of the selected image frame sequence. When the tail image frame of the selected image frame sequence is closer to the head image frame of the divided scene than the tail image frame of the divided scene at the position of the divided scene, the head image frame of the divided scene can be used as the tail image frame of the selected image frame sequence. When the tail image frame of the selected image frame is closer to the tail image frame of the divided scene than the head image frame of the divided scene at the position of the divided scene, the tail image frame of the divided scene can be used as the tail image frame of the selected image frame sequence.
[0181] In the above implementation, a preset range of the number of image frames and a range of the number of image-caption combinations are determined, and the video file is segmented according to the range of the number of image frames and the range of the number of image-caption combinations to obtain video clips. By determining the preset range of the number of image frames, the video clips can be controlled within the range of telling a short plot without making the semantic information too monotonous. By determining the range of the number of image-caption combinations, it can be ensured that the semantics are not too sparse.
[0182] This specification provides a method for processing video files. Figure 8 , the method may include the following steps:
[0183] S810: Obtain an image-subtitle combination corresponding to the video file.
[0184] The video file is divided into a number of unit segments; the image-caption combination is obtained based on the image frames in the video file and the caption texts aligned with the image frames in time.
[0185] Specifically, a video file may include a subtitle file and an image sequence. The subtitle file and the image frames included in the image sequence may be aligned in the time sequence in combination with the frame rate and the timestamp of the subtitle file. Some image frames may be matched with subtitle texts aligned with their time, while some image frames may not be matched with letter texts aligned with their time. Based on the subtitle texts matched with their time, an image subtitle combination may be generated.
[0186] S820: Segment the video file according to a preset video segmentation condition to obtain a video clip including at least one unit segment.
[0187] Among them, the video clip corresponds to a target image subtitle combination; the unit segment to which the target image frame included in the target image subtitle combination belongs is the target unit segment; the target image subtitle combination corresponds to hierarchical marking information, and the hierarchical marking information is used to represent the hierarchical structure of the target image subtitle combination in the video file; the hierarchical marking information is generated based on the segment identifier of the target unit segment.
[0188] Specifically, an image frame is selected from the beginning of the video file. When the selected image frame meets the preset video segmentation condition, the image frame sequence meeting the preset video segmentation condition may be segmented to obtain a video clip containing at least one unit segment.
[0189] The above-mentioned video file processing method obtains the image-subtitle combination corresponding to the video file, segments the video file to obtain a video clip including at least one unit segment, and generates hierarchical marking information of the target image-subtitle combination based on the segment identifier of the target unit segment. By generating the hierarchical marking information of the target image-subtitle combination, the annotation of the data set samples constructed based on the video file can be enriched. The visual narrative structure of the video clip can be better determined through the constructed data sample set, thereby understanding the overall content of the video clip.
[0190] This specification provides a visual narrative reasoning model training method. Fig. 9 , the method may include the following steps:
[0191] S910: Obtain a data set sample constructed based on the method of any of the above implementations.
[0192] S920: Use the data set samples to train the initial visual narrative reasoning model until the model training stop condition is met to obtain the target visual narrative reasoning model.
[0193] In some cases, key visual or auditory information can be inferred to form a generalized visual narrative. The entire data set can be divided into four parts: training set, validation set, internal domain test set and external domain test set. The data in the validation set and the internal domain test set can come from the same movie set as the training set, and the validation set and the internal domain test set can be selected by selecting clips at equal intervals based on the movie clip numbers of the training set. The external domain test set can come from a different movie set from the training set, and the external domain test set can be a clip selected from a movie that does not belong to the training set but only has its pre-extracted visual features. Exemplarily, the training set, validation set, internal domain test set and external domain test set can account for 70%, 6%, 12% and 12% of the total data set, respectively. Dividing the data set into training set, validation set, internal domain test set and external domain test set can help evaluate the generalization ability of the model.
[0194] Specifically, the image subtitle combination obtained by the method of any of the above-mentioned embodiments can be used to construct a data set sample, and the data set sample can be rearranged, and the rearranged data set sample can improve the accuracy of the model. The initial visual narrative reasoning model is trained using the rearranged data set samples. During the model training process, the model loss value is determined, and the initial visual narrative reasoning is updated based on the model loss value. Similarly, the updated initial visual narrative reasoning continues to be trained, and when the model training stopping condition is reached, the target visual narrative reasoning model can be obtained. Among them, the model training stopping condition can be that the model loss value tends to converge, or that the training rounds reach a preset number of rounds.
[0195] In the above implementation, a data set sample constructed based on the method of any of the above implementations is obtained, and the initial visual narrative reasoning is trained using the data set sample until the model training stop condition is met to obtain the target visual narrative reasoning. The obtained target visual narrative reasoning can be used to predict the visual narrative structure and content of the video segment, and the content of the movie can be understood. The trained target visual narrative reasoning model can be applied to a variety of downstream tasks.
[0196] In some implementations, the visual narrative reasoning model is any one of a video model set, which includes an action prediction model, a video question-answering model, and a video-based dialogue generation model.
[0197] In some embodiments, the following criteria can be used to select movie sources and scripts related to the selected movie sources from the MovieNet dataset to create a dataset, which can be named MoviePuzzle. The movie screen should be a color image, and the picture quality of the movie should not be too dark or blurry. The movie should have English subtitles, but some movies in the original MovieNet dataset lack subtitles or the subtitles are in other languages. Movies with subtitles in other languages can be translated into English subtitles. The movie should have a plot synopsis, which can help downstream tasks complete the task of associating the movie plot. However, the labels in the original MovieNet dataset need to be further enriched to better determine the visual narrative structure of the video clips.
[0198] Each image frame can be matched with its corresponding subtitle using the following steps: First, most of the narration and background sounds can be removed from the subtitles, and non-English words can be replaced with English letters. Second, each dialogue text, i.e., subtitle, is only paired with images that fall completely within its corresponding time period, rather than images within the closest time period. Finally, in the case where a dialogue corresponds to multiple image frames, these image frames can be merged, or the middle image frame can be selected as the most representative matching image. The following conditions are used as references to segment the movie into video clips, where the text subtitles are aligned by image frames, and the length of each video clip can be limited to 10 to 20 frames. The image frames containing subtitles in each video clip must account for more than 80% of all image frames. A greedy algorithm can be used for movie segmentation to obtain video clips. After obtaining the video clips, the data set can be divided into a training set, a validation set, an in-domain test set, and an out-of-domain test set according to the ratios of 70%, 6%, 12%, and 12%.
[0199] This specification provides a data set sample construction device 1000, see Fig.10 The data set sample construction device 1000 includes: an image subtitle combination acquisition module 1010, a video segmentation module 1020, a tag information generation module 1030, and a data set sample construction module 1040.
[0200] The image-caption combination acquisition module 1010 is used to acquire an image-caption combination corresponding to a video file; wherein the video file is divided into a plurality of unit segments; the image-caption combination is obtained based on an image frame in the video file and / or a caption text aligned with the image frame in time;
[0201] The video segmentation module 1020 is used to segment the video file to obtain a video segment including at least one unit segment; wherein the video segment corresponds to a target image-caption combination; the unit segment to which the target image frame included in the target image-caption combination belongs is the target unit segment;
[0202] The tag information generating module 1030 is used to generate hierarchical tag information of the target image-subtitle combination based on the segment identifier of the target unit segment; wherein the hierarchical tag information is used to represent the hierarchical structure of the target image-subtitle combination in the video file;
[0203] The data set sample construction module 1040 is used to construct the data set sample based on the hierarchical tag information of the video clip and the target image subtitle combination.
[0204] This specification embodiment provides a video file processing device 1100, see Fig.11The video file processing device 1100 includes: an image subtitle combination acquisition module 1110 and a video segmentation module 1120.
[0205] The image-caption combination acquisition module 1110 is used to acquire the image-caption combination corresponding to the video file; wherein the video file is divided into a plurality of unit segments; the image-caption combination is obtained based on the image frames in the video file and the caption texts aligned with the image frames in time;
[0206] The video segmentation module 1120 is used to segment the video file according to a preset video segmentation condition to obtain a video clip including at least one unit segment; wherein the video clip corresponds to a target image-subtitle combination; the unit segment to which the target image frame included in the target image-subtitle combination belongs is the target unit segment; the target image-subtitle combination corresponds to hierarchical marking information, and the hierarchical marking information is used to characterize the hierarchical structure of the target image-subtitle combination in the video file; the hierarchical marking information is generated based on the segment identifier of the target unit segment.
[0207] This specification provides a visual narrative reasoning model training device 1200, see Fig.12 The visual narrative reasoning model training device 1200 includes: a data set sample construction module 1210 and a visual narrative reasoning model training module 1220.
[0208] A data set sample construction module 1210, used to obtain a data set sample constructed based on the method described in any of the above embodiments;
[0209] The visual narrative reasoning model training module 1220 is used to train the initial visual narrative reasoning model using the data set samples until the model training stop condition is met to obtain a target visual narrative reasoning model.
[0210] For a detailed description of the data set sample construction device, the video file processing device, and the visual narrative reasoning model training device, please refer to the description of the data set sample construction method, the video file processing method, and the visual narrative reasoning model training method above, which will not be repeated here.
[0211] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.13As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for constructing a data set sample is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0212] Those skilled in the art will understand that Fig.13 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution disclosed in this specification, and does not constitute a limitation on the computer device to which the solution disclosed in this specification is applied. Specifically, the computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0213] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method steps in the above embodiments when executing the computer program.
[0214] An embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.
[0215] One embodiment of the present specification provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.
[0216] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
Claims
1. A method for constructing a data set sample, characterized in that: The method comprises: Acquire an image-caption combination corresponding to a video file; wherein the video file is divided into a plurality of unit segments; the image-caption combination is obtained based on an image frame in the video file and / or a caption text aligned with the image frame in time; Segmenting the video file to obtain a video segment including at least one unit segment; wherein the video segment corresponds to a target image-caption combination; and the unit segment to which the target image frame included in the target image-caption combination belongs is the target unit segment; Generating hierarchical marking information of the target image-subtitle combination based on the segment identifier of the target unit segment; wherein the hierarchical marking information is used to represent the hierarchical structure of the target image-subtitle combination in the video file; The dataset sample is constructed based on the hierarchical tag information of the combination of the video clip and the target image caption.
2. The method according to claim 1, characterized in that The unit segment is a scene, and the unit segment to which the target image frame included in the target image subtitle combination belongs is a target scene; The step of generating hierarchical marking information of the target image subtitle combination based on the segment identifier of the target unit segment includes: The hierarchical marking information of the target image-caption combination is generated based on the scene identification of the target scene.
3. The method according to claim 1, characterized in that The unit segment is a storyboard, and the unit segment to which the target image frame included in the target image-subtitle combination belongs is the target storyboard; and the hierarchical marking information of the target image-subtitle combination is generated based on the segment identifier of the target unit segment, including: The hierarchical marking information of the target image-caption combination is generated based on the storyboard identifier of the target storyboard.
4. The method according to claim 1, characterized in that The unit segment includes a scene and a storyboard, the storyboard to which the target image frame included in the target image-caption combination belongs is the target storyboard, and the scene to which the target storyboard belongs is the target scene; the hierarchical marking information of the target image-caption combination is generated based on the segment identifier of the target unit segment, including: The hierarchical marking information of the target image-caption combination is generated based on the storyboard identifier of the target storyboard and the scene identifier of the target scene.
5. The method according to claim 1, characterized in that The video file includes a first scene, and the method further includes: Determining an initial scene boundary frame of the first scene and a storyboard boundary frame of the storyboard; If the initial scene boundary frame of the first scene is not a storyboard boundary frame of any storyboard, determining the any storyboard as a divided storyboard; Boundary alignment processing is performed based on the storyboard boundary frames of the divided storyboards and the initial scene boundary frame of the first scene to obtain a target scene boundary frame of the first scene; wherein the target scene boundary frame is used as a reference for segmenting the video file.
6. The method according to claim 5, characterized in that The video file includes a second scene adjacent to the first scene; the boundary alignment process is performed based on the storyboard boundary frames of the divided storyboards and the initial scene boundary frame of the first scene to obtain a target scene boundary frame of the first scene; Determine a first matching condition between the divided storyboard and the first scene, and a second matching condition between the divided storyboard and the second scene; Determining, according to the first matching situation and the second matching situation, a target matching scene matching the divided storyboard in the first scene and the second scene; The target scene boundary frame is set according to the storyboard boundary frame of the divided storyboard to include the divided storyboard in the target matching scene.
7. The method according to claim 6, characterized in that The first scene is earlier than the second scene, and if the target matching scene is the first scene, setting the tail scene boundary frame of the divided storyboard to the tail scene boundary frame of the first scene, wherein the tail scene boundary frame of the first scene is the target scene boundary frame; or The first scene is earlier than the second scene. If the target matching scene is the second scene, the head scene boundary frame of the divided scene is set to the head scene boundary frame of the second scene, wherein the head scene boundary frame of the second scene is the target scene boundary frame.
8. The method according to claim 1, characterized in that The step of generating the hierarchical marking information of the target image subtitle combination based on the segment identifier of the target unit segment includes at least one of the following situations: If the unit segment is a storyboard, the storyboard identifier of the target storyboard is set to the frame storyboard identifier of the target image frame as the hierarchical marking information; wherein the target storyboard is the unit segment to which the target image frame included in the target image subtitle combination belongs; If the unit segment is a scene, the scene identifier of the target scene is set as the frame scene identifier of the target image frame as the hierarchical marking information; wherein the target scene is the unit segment to which the target image frame included in the target image subtitle combination belongs; If the unit segment includes a scene and a storyboard, the storyboard scene marking information of the target image subtitle combination is generated according to the frame storyboard identifier and the frame scene identifier as the hierarchical marking information.
9. The method according to claim 1, characterized in that: The method for generating the image subtitle combination includes: Based on the subtitle file and the image sequence included in the video file, alignment processing is performed on the time sequence to obtain time-aligned image frames and subtitle texts; The image-caption combination is generated based on the time-aligned image frames and caption texts.
10. The method according to claim 9, characterized in that The subtitle file corresponds to a plurality of subtitle texts; the subtitle file and the image sequence included in the video file are aligned in time sequence to obtain time-aligned image frames and subtitle texts, including: In the image sequence, determining a matching image frame located in a time period corresponding to each subtitle text; The time-aligned image frame and subtitle text are determined based on each subtitle text and the matching image frame.
11. The method according to claim 10, characterized in that The determining the time-aligned image frame and subtitle text based on each subtitle text and the matching image frame comprises: If there are multiple matching image frames, screening and / or merging the matching image frames to obtain a representative image frame matching each subtitle text; The time-aligned image frame and subtitle text are determined based on each subtitle text and the representative image frame.
12. The method according to claim 1, characterized in that The method further comprises: In the process of segmenting the video file, for a first image frame sequence that does not match a subtitle text, if the number of frames of the first image frame sequence meets a preset frame discarding condition, the first image frame sequence is discarded.
13. The method according to claim 1, characterized in that The step of segmenting the video file to obtain a video clip including at least one unit segment includes: Determine a preset range of image frame quantity and image subtitle combination quantity range; The video file is segmented according to the image frame quantity range and the image subtitle combination quantity range to obtain the video segments.
14. A video file processing method, characterized in that: The method comprises: Acquire an image-subtitle combination corresponding to the video file; wherein the video file is divided into a plurality of unit segments; the image-subtitle combination is obtained based on an image frame in the video file and a subtitle text time-aligned with the image frame; The video file is segmented according to a preset video segmentation condition to obtain a video clip including at least one unit segment; wherein the video clip corresponds to a target image-subtitle combination; the unit segment to which the target image frame included in the target image-subtitle combination belongs is the target unit segment; the target image-subtitle combination corresponds to hierarchical marking information, and the hierarchical marking information is used to characterize the hierarchical structure of the target image-subtitle combination in the video file; the hierarchical marking information is generated based on the segment identifier of the target unit segment.
15. A visual narrative reasoning model training method, characterized in that: The method comprises: Obtaining a data set sample constructed based on the method of any one of claims 1 to 14; The initial visual narrative reasoning model is trained using the data set samples until the model training stop condition is met, thereby obtaining a target visual narrative reasoning model.
16. The method according to claim 15, characterized in that The visual narrative reasoning model is any one of a video model set; the video model set includes an action prediction model, a video question-answering model, and a video-based dialogue generation model.
17. A data set sample construction device, characterized in that: The device comprises: An image-caption combination acquisition module is used to acquire an image-caption combination corresponding to a video file; wherein the video file is divided into a plurality of unit segments; the image-caption combination is obtained based on an image frame in the video file and / or a caption text aligned with the image frame in time; A video segmentation module is used to segment the video file to obtain a video segment including at least one unit segment; wherein the video segment corresponds to a target image-caption combination; the unit segment to which the target image frame included in the target image-caption combination belongs is the target unit segment; A marking information generating module, used to generate hierarchical marking information of the target image-subtitle combination based on the segment identifier of the target unit segment; wherein the hierarchical marking information is used to represent the hierarchical structure of the target image-subtitle combination in the video file; The data set sample construction module is used to construct the data set sample based on the hierarchical labeling information of the video clip and the target image subtitle combination.
18. A video file processing device, characterized in that: The device comprises: An image-caption combination acquisition module is used to acquire an image-caption combination corresponding to the video file; wherein the video file is divided into a plurality of unit segments; the image-caption combination is obtained based on an image frame in the video file and a caption text aligned with the image frame in time; A video segmentation module is used to segment the video file according to a preset video segmentation condition to obtain a video clip including at least one unit segment; wherein the video clip corresponds to a target image-subtitle combination; the unit segment to which the target image frame included in the target image-subtitle combination belongs is the target unit segment; the target image-subtitle combination corresponds to hierarchical marking information, and the hierarchical marking information is used to characterize the hierarchical structure of the target image-subtitle combination in the video file; the hierarchical marking information is generated based on the segmentation identifier of the target unit segment.
19. A visual narrative reasoning model training device, characterized in that: The device comprises: A data set sample construction module, used to obtain a data set sample constructed based on the method according to any one of claims 1 to 14; The visual narrative reasoning model training module is used to train the initial visual narrative reasoning model using the data set samples until the model training stop condition is met to obtain the target visual narrative reasoning model.
20. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 16 are implemented.
21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.
Citation Information
Cited By
Video processing method, visual analysis model training method and electronic equipment
CN120431503A
