Video scene segmentation method and device, electronic equipment and storage medium
By combining video files and subtitle files to generate film reels, the problem of visual information not being related to the plot logic is solved, enabling accurate judgment of transitions between scenes with non-drastic changes and improving the accuracy of video scene segmentation.
Patent Information
- Application Number
- CN202510809167.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-17
- Publication Date
- 2025-10-28
AI Technical Summary
Existing video scene segmentation methods fail to effectively link visual information with plot logic, resulting in an inability to accurately capture scene transitions that do not involve drastic changes.
By acquiring the video file and subtitle file to be segmented, scene segmentation is performed to generate an initial scene segmentation file. Keyframes are selected to generate a film reel, and scene segmentation is performed based on the storage location of the film reel. Accurate segmentation is then achieved by combining a large language model.
It enables accurate judgment of transitions in non-drastic scene changes, improving the accuracy and effectiveness of video scene segmentation.
Smart Images
Figure CN120856950A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for segmenting video scenes. Background Technology
[0002] In recent years, with the continuous development of network technology, the film and television industry, including short dramas, web dramas, and short videos, has entered a period of explosive growth in content. The massive amount of video data generated every day has not only grown exponentially in scale, but has also reached new heights in terms of content diversity and production complexity. This has posed unprecedented challenges to the efficient analysis and processing technology of video data.
[0003] In related technologies, existing video scene segmentation methods mainly focus on visual feature analysis. Although this method can segment video scenes to a certain extent, it fails to accurately capture the connotation of the scene because it does not effectively associate visual information with the plot logic. Consequently, it cannot accurately judge non-drastic scene transitions, such as life scenes and gradual transitions. Summary of the Invention
[0004] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for segmenting video scenes, in order to solve the technical problem that the inability to accurately capture the scene's meaning and thus accurately judge non-drastic scene transitions is due to the failure to effectively associate visual information with plot logic. The specific technical solution is as follows:
[0005] In a first aspect of this application, a method for segmenting a video scene is provided, the method comprising:
[0006] Obtain the video file to be segmented and its corresponding subtitle file;
[0007] The subtitle file is segmented into scenes to obtain an initial scene segmentation file. The initial scene segmentation file contains N dialogue segments, each dialogue segment corresponding to a scene, where N is a positive integer.
[0008] For any of the dialogue segments, select M keyframes from the video file to be segmented based on the dialogue segment to generate a film reel, where M is a positive integer;
[0009] Determine the storage location of the film image corresponding to the dialogue segment in the initial scene segmentation file, and store the film image in the initial scene segmentation file according to the storage location of the film image to obtain the candidate scene segmentation file;
[0010] Based on the candidate scene segmentation file, the video file to be segmented is segmented into scenes.
[0011] In an optional implementation, the step of segmenting the subtitle file into scenes to obtain an initial scene segmentation file includes:
[0012] Determine the time interval between adjacent texts in the subtitle file; when the time interval is greater than a preset time threshold, set a scene segmentation point between adjacent texts.
[0013] And / or,
[0014] Determine the semantic similarity between adjacent texts in the subtitle file. When the semantic similarity is lower than a preset semantic threshold, set scene segmentation points between adjacent texts.
[0015] The subtitle file is segmented into N dialogue segments based on the scene segmentation points.
[0016] For any of the dialogue segments, the dialogue segments are analyzed to obtain the corresponding scene summary and dialogue time period;
[0017] The initial scene segmentation file is generated based on the N dialogue segments, their corresponding scene summaries, and the dialogue time periods.
[0018] In an optional implementation, the step of selecting M keyframes from the video file to be segmented based on the dialogue segment and generating a film reel includes:
[0019] Based on the dialogue time segment corresponding to the dialogue segment, determine the video segment from the video file to be segmented;
[0020] Select M keyframes from the video segment;
[0021] The film image is generated based on the M keyframes.
[0022] In an optional implementation, generating the film image based on the M keyframes includes:
[0023] The M keyframes are sorted by time to generate a keyframe sequence;
[0024] Size normalization is performed on each keyframe in the keyframe sequence;
[0025] All keyframes in the keyframe sequence after size normalization are stitched together to generate the film image.
[0026] In an optional implementation, determining the storage location of the film reel corresponding to the dialogue segment in the initial scene segmentation file includes:
[0027] The node information of the dialogue segment is determined in the initial scene segmentation file;
[0028] Based on the node information of the dialogue segment, a corresponding storage node is created, and the storage node is used as the storage location.
[0029] In an optional implementation, the step of storing the film image in the initial scene segmentation file according to the storage location of the film image to obtain a candidate scene segmentation file includes:
[0030] Generate film image links based on the film images;
[0031] The film image is linked and stored in the location where the film image is stored in the initial scene segmentation file to obtain the candidate scene segmentation file.
[0032] In an optional implementation, the step of segmenting the video file to be segmented according to the candidate scene segmentation file includes:
[0033] Based on the candidate scene segmentation files and scene segmentation instructions, construct prompt words for the large language model;
[0034] The prompt words are input into a large language model to obtain structured scene data, which contains G scene information, where G is a positive integer;
[0035] Based on any of the scene information, the video file to be segmented is segmented into scenes.
[0036] In a second aspect of this application, a video scene segmentation apparatus is also provided, the apparatus comprising:
[0037] The acquisition module retrieves the video file to be segmented and its corresponding subtitle file.
[0038] The initial segmentation module performs scene segmentation on the subtitle file to obtain an initial scene segmentation file. The initial scene segmentation file contains N dialogue segments, each dialogue segment corresponding to a scene, where N is a positive integer.
[0039] The extraction module, for any one of the dialogue segments, selects M keyframes from the video file to be segmented based on the dialogue segment and generates a film reel, where M is a positive integer;
[0040] The first processing module determines the storage location of the film image corresponding to the dialogue segment in the initial scene segmentation file, and stores the film image in the initial scene segmentation file according to the storage location of the film image to obtain the candidate scene segmentation file;
[0041] The second processing module performs scene segmentation on the video file to be segmented based on the candidate scene segmentation file.
[0042] In a third aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0043] Memory, used to store computer programs;
[0044] The processor, when executing a program stored in memory, implements the video scene segmentation method described in any one of the first aspects above.
[0045] In a fourth aspect of the embodiments of this application, a storage medium is also provided, wherein the storage medium stores instructions that, when run on a computer, cause the computer to execute the video scene segmentation method described in any of the first aspects above.
[0046] In a fifth aspect of the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the video scene segmentation method described in any of the first aspects above.
[0047] The technical solution provided in this application involves obtaining a video file to be segmented and its corresponding subtitle file, performing scene segmentation on the subtitle file to obtain an initial scene segmentation file. This initial scene segmentation file contains N dialogue segments, each corresponding to a scene, where N is a positive integer. For any dialogue segment, M keyframes are selected from the video file to be segmented based on the dialogue segment, generating a film reel image, where M is a positive integer. The storage location of the film reel image corresponding to the dialogue segment in the initial scene segmentation file is determined, and the film reel image is stored in the initial scene segmentation file according to its location, resulting in a candidate scene segmentation file. Based on the candidate scene segmentation file, scene segmentation is performed on the video file to be segmented. This combination of the subtitle file and the video file to be segmented solves the technical problem that, due to the lack of effective association between visual information and plot logic, it is impossible to accurately capture the scene's connotation, and thus impossible to accurately judge non-drastic scene transitions. Attached Figure Description
[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0050] Figure 1 A schematic diagram illustrating the implementation process of a video scene segmentation method provided in this application embodiment;
[0051] Figure 2 A schematic diagram illustrating the implementation process of another video scene segmentation method provided in this application embodiment;
[0052] Figure 3 A schematic diagram illustrating the implementation process of a scene segmentation method provided in this application embodiment;
[0053] Figure 4 This is a schematic diagram illustrating the implementation process of a film image generation method provided in this application embodiment;
[0054] Figure 5 A schematic diagram illustrating the implementation process of a method for generating candidate scene segmentation files provided in this application embodiment;
[0055] Figure 6 A schematic diagram of the structure of a video scene segmentation device provided in an embodiment of this application;
[0056] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0058] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.
[0059] like Figure 1 The diagram shown illustrates the implementation flow of a video scene segmentation method provided in this application embodiment. The method specifically includes the following steps:
[0060] S101, Obtain the video file to be segmented and the corresponding subtitle file.
[0061] In this embodiment of the application, the video file to be segmented and the corresponding subtitle file are obtained. The format of the video file to be segmented can be MP4, AVI, MOV, etc., and contains information such as actual picture and sound content. The format of the subtitle file can be SRT, ASS, etc., and contains the subtitle text content (dialogue, narration, etc.) of the video to be segmented and the corresponding timestamp. This application example does not limit this.
[0062] For example, if you want to segment movie A into scenes, you need to obtain the video file to be segmented, "movieA.mp4", and the corresponding subtitle file, "movieA.srt".
[0063] S102, perform scene segmentation on the subtitle file to obtain an initial scene segmentation file. The initial scene segmentation file contains N dialogue segments, each dialogue segment corresponds to a scene, and N is a positive integer.
[0064] In this embodiment, the subtitle file is segmented into scenes to obtain an initial scene segmentation file. Scene segmentation can be based on information such as text content and timestamps in the subtitle file. By analyzing the subtitle text, transition points between different scenes (such as changes in dialogue theme, character switching, plot twists, etc.) are identified, thus obtaining the initial scene segmentation file. For example, when the dialogue suddenly changes from introducing the movie background to a conversation between characters, it may mean a scene transition; or when a character switches from one person to another and the plot develops anew, this can also serve as a basis for scene transitions. This embodiment does not limit this approach.
[0065] The initial scene segmentation file mentioned above can contain N dialogue segments, each dialogue segment corresponding to a scene. The format of the initial scene segmentation file can be an easy-to-parse and process format, such as JSON format, containing information such as the start time, end time, and corresponding dialogue segment of each scene. This application example does not limit this.
[0066] For example, analyzing the subtitle file "Movie A.srt", we find that the text content during the time period 00:00:00,000-00:00:10,000 mainly revolves around the movie's opening introduction and the appearance of the main characters. Therefore, this time period is designated as the first scene. During the time period 00:00:10,000-00:00:18,000, the dialogue involves the initial conversations between characters and the development of the plot; this time period is designated as the second scene (Scene 2). The information for these two scenes, including start and end times and dialogue excerpts, is stored in the initial scene division file in JSON format. The specific storage format can be as follows:
[0067]
[0068] S103: For any dialogue segment, select M keyframes from the video file to be segmented based on the dialogue segment and generate a film reel, where M is a positive integer.
[0069] In this embodiment, for any dialogue segment, M keyframes (e.g., 20 keyframes, 30 keyframes, etc.) are selected from the video file to be segmented to generate a film reel, where M is a positive integer. A keyframe is a frame that represents the visual characteristics of the corresponding scene. M keyframes can be selected from the video file to be segmented at fixed time intervals, or based on significant changes in the visual content (e.g., scene transitions, large changes in character movements, etc.). This embodiment does not limit the selection in this way.
[0070] For example, for scene A (00:00:00,000-00:02:30,000), 15 keyframes were selected from the video file "Movie A.mp4" at a rate of one frame every 10 seconds, thus generating a film reel.
[0071] S104. Determine the storage location of the film images corresponding to the dialogue segments in the initial scene segmentation file, and store the film images in the initial scene segmentation file according to the storage location of the film images to obtain the candidate scene segmentation file.
[0072] In this embodiment, the storage location of the film reel corresponding to the dialogue segment in the initial scene segmentation file is determined, and the film reel is stored in the initial scene segmentation file according to its storage location, resulting in a candidate scene segmentation file. Specifically, a field can be added to each scene in the initial scene segmentation file to store the path or data of the film reel corresponding to that scene. The path or data of the film reel is then stored in the initial scene segmentation file according to its storage location, resulting in a candidate scene segmentation file. The candidate scene segmentation file may contain the scene's time information, dialogue segments, and the film reel corresponding to each scene.
[0073] For example, in the "Initial Scene Division.json" file, add a "Film Image Path" field for each scene. For the first scene, save the generated film image as "Scene 1 Film Image.jpg" and store its path " / path / to / Scene 1 Film Image.jpg" in the "Film Image Path" field. Process other scenes in the same way to obtain the candidate scene division file "Candidate Scene Division.json".
[0074] S105, divide the file according to the candidate scene, and perform scene segmentation on the video file to be segmented.
[0075] In this embodiment of the application, the video file to be segmented is segmented according to the candidate scene segmentation file. The candidate scene information can be input into the large model. The large model analyzes the scene time information, dialogue segments and film images in the candidate scene segmentation file, identifies the boundaries and features between scenes, and performs scene segmentation on the video file to be segmented based on the analysis results.
[0076] Based on the above description of the technical solution provided in the embodiments of this application, a video file to be segmented and a corresponding subtitle file are obtained. The subtitle file is then segmented into scenes to obtain an initial scene segmentation file. The initial scene segmentation file contains N dialogue segments, each dialogue segment corresponding to a scene, where N is a positive integer. For any dialogue segment, M keyframes are selected from the video file to be segmented based on the dialogue segment to generate a film reel image, where M is a positive integer. The storage location of the film reel image corresponding to the dialogue segment in the initial scene segmentation file is determined. According to the storage location of the film reel image, it is stored in the initial scene segmentation file to obtain a candidate scene segmentation file. Based on the candidate scene segmentation file, scene segmentation is performed on the video file to be segmented. By combining the subtitle file with the video file to be segmented in this way, the technical problem of not being able to accurately capture the scene's connotation due to the lack of effective association between visual information and plot logic, and thus being unable to accurately judge non-drastic scene transitions, can be solved.
[0077] like Figure 2 The diagram shown illustrates the implementation flow of another video scene segmentation method provided in this application embodiment, which may specifically include the following:
[0078] S201, Obtain the video file to be segmented and the corresponding subtitle file.
[0079] In this embodiment of the application, the video file to be segmented and the corresponding subtitle file are obtained. The format of the video file to be segmented can be MP4, AVI, MOV, etc., and contains information such as actual picture and sound content. The format of the subtitle file can be SRT, ASS, etc., and contains the subtitle text content (dialogue, narration, etc.) of the video to be segmented and the corresponding timestamp. This application example does not limit this.
[0080] For example, if you want to segment movie A into scenes, you need to obtain the video file to be segmented, "movieA.mp4", and the corresponding subtitle file, "movieA.srt".
[0081] S202, Perform scene segmentation on the subtitle file to obtain an initial scene segmentation file. The initial scene segmentation file contains N dialogue segments, each dialogue segment corresponds to a scene, and N is a positive integer.
[0082] In this embodiment, the subtitle file is segmented into scenes to obtain an initial scene segmentation file. Scene segmentation can be based on information such as text content and timestamps in the subtitle file. By analyzing the subtitle text, transition points between different scenes (such as changes in the theme of dialogue, character changes, plot twists, etc.) are identified, thereby obtaining the initial scene segmentation file. For example, when the dialogue suddenly changes from introducing the movie background to a conversation between characters, it may mean that a scene has changed; or when a character switches from one person to another and the plot develops anew, this can also serve as a basis for scene transition. This embodiment does not limit this.
[0083] The initial scene segmentation file mentioned above can contain N dialogue segments, each dialogue segment corresponding to a scene. The format of the initial scene segmentation file can be an easy-to-parse and process format, such as JSON format, which contains information such as the start time, end time, and corresponding dialogue segment of each scene. This application example does not limit this.
[0084] For example, analyzing the subtitle file "Movie A.srt", we find that the text content during the time period 00:00:00,000-00:00:10,000 mainly revolves around the movie's opening introduction and the appearance of the main characters. Therefore, this time period is designated as the first scene. During the time period 00:00:10,000-00:00:18,000, the dialogue involves the initial conversations between characters and the development of the plot; this time period is designated as the second scene (Scene 2). The information for these two scenes, including start and end times and dialogue excerpts, is stored in the initial scene division file in JSON format. The specific storage format can be as follows:
[0085]
[0086]
[0087] For details on how to segment the subtitle file into scenes to obtain the initial scene segmentation file, please refer to... Figure 3 The method shown. (As illustrated) Figure 3 The diagram shown illustrates the implementation process of a scene segmentation method provided in this application embodiment, which may specifically include the following steps:
[0088] S301, determine the time interval between adjacent texts in the subtitle file, and when the time interval is greater than a preset time threshold, set a scene segmentation point between adjacent texts, and / or, determine the semantic similarity between adjacent texts in the subtitle file, and when the semantic similarity is lower than a preset semantic threshold, set a scene segmentation point between adjacent texts.
[0089] In this embodiment, the time interval between adjacent text in the subtitle file is determined. When the time interval is greater than a preset time threshold, a scene segmentation point is set between the adjacent text. And / or, the semantic similarity between adjacent text in the subtitle file is determined. When the semantic similarity is lower than a preset semantic threshold, a scene segmentation point is set between the adjacent text. This can be achieved by traversing the subtitle file and calculating the time interval between adjacent text. A suitable natural language processing algorithm (such as cosine similarity, Jaccard similarity, etc.) can be selected to calculate the semantic similarity between adjacent text, thereby determining the semantic similarity between adjacent text in the subtitle file. This embodiment does not limit this approach.
[0090] For example, suppose the preset time threshold is 5 seconds and the preset semantic similarity threshold is 0.7. For adjacent text in the subtitle file: Text 1: "Hello, the weather is nice today." (timestamp: 00:00:00) Text 2: "Yes, let's go for a walk." (timestamp: 00:00:18), since the time interval between Text 1 and Text 2 is 18 seconds, which is greater than the preset time threshold of 5 seconds, a scene split point can be set between Text 1 and Text 2.
[0091] For example, consider another pair of adjacent texts: Text 3: "Did you like this movie?" (timestamp: 00:02:00) Text 4: "I think the plot is just so-so." (timestamp: 00:02:10). Using a semantic similarity algorithm, we calculate the similarity between Text 3 and Text 4. Assuming the result is 0.6, which is lower than the preset semantic similarity threshold of 0.7, we also set a scene segmentation point between Text 3 and Text 4.
[0092] It should be noted that the preset time threshold can be set manually according to business needs, such as 10s, 20s, etc. The preset semantic similarity threshold can be the average similarity threshold obtained based on the time interval between all adjacent texts in the entire subtitle file, or it can be set manually according to business needs, such as 0.7, 0.5, etc. In addition, for faster-paced dramas, a lower time threshold and a higher semantic similarity threshold can be set; for slower-paced dramas, a higher time threshold and a lower semantic similarity threshold can be set. This application example does not limit this.
[0093] S302, perform scene segmentation on the subtitle file according to the scene segmentation points to obtain N dialogue segments.
[0094] In this embodiment of the application, the subtitle file is segmented into N dialogue segments based on scene segmentation points, and each dialogue segment corresponds to a scene.
[0095] For example, if scene breakpoints are set between text 1 and text 2, between text 2 and text 3, and between text 5 and text 6, then the resulting dialogue segments are: Segment 1: Text 1; Segment 2: Text 2; Segment 3: Text 3, Text 4, Text 5; Segment 4: Text 6.
[0096] S303: For any dialogue segment, analyze the dialogue segment to obtain the corresponding scene summary and dialogue time period.
[0097] In this embodiment, for any dialogue segment, the dialogue segment is analyzed to obtain a corresponding scene summary and dialogue time period. The main content or plot of the dialogue segment, i.e., the scene summary, is obtained by extracting the summary from the dialogue segment. The corresponding dialogue start time and dialogue end time are obtained by analyzing each dialogue segment. Based on the dialogue start time and dialogue end time, the dialogue time period is determined. This embodiment does not limit the scope of this embodiment.
[0098] For example, by analyzing the text content of dialogue segment 1, we obtain the scene summary of segment 1: "Character A and Character B greet each other at the beginning and talk about the weather." Similarly, the dialogue start time is 00:00:00 and the dialogue end time is 00:00:18. Therefore, the dialogue time period of segment 1 is "00:00:00-00:00:18". Similarly, we obtain the scene summary of segment 2: "Character B suggests going for a walk." The dialogue time period of segment 2 is "00:00:18-00:00:28" (assuming the end time of text 2 is 00:00:28). The scene summary of segment 3: "Character A asks Character B for his opinion on the movie, and Character B says the plot is so-so." The dialogue time period of segment 3 is "00:02:00-00:02:20" (assuming the end time of text 4 is 00:02:20).
[0099] S304. Generate an initial scene segmentation file based on N dialogue segments and their corresponding scene summaries and dialogue time periods.
[0100] In this embodiment, an initial scene segmentation file is generated based on N dialogue segments and their corresponding scene summaries and dialogue time periods. The N dialogue segments and their corresponding scene summaries and dialogue time periods can be organized into structured data in JSON or other formats to generate the initial scene segmentation file. The initial scene segmentation file contains key fields such as scene segmentation, scene summary, dialogue time period, and dialogue content. In addition, the generated initial scene segmentation file can be verified to ensure the accuracy and completeness of the information. This application example does not limit this aspect.
[0101] For example, the format of the final initial scene partitioning file can be:
[0102]
[0103]
[0104] S203: For any dialogue segment, determine the video segment from the video file to be segmented based on the dialogue time period corresponding to the dialogue segment.
[0105] In this embodiment of the application, for any dialogue segment, the corresponding dialogue time period is obtained from the initial scene segmentation file. For example, the dialogue time period can be determined based on the start and end times of each scene in the initial scene segmentation file, or the dialogue time period field information of the initial scene segmentation file can be directly obtained to determine the dialogue time period. Based on the dialogue time period, the corresponding video segment is located and extracted from the video file to be segmented. Video processing tools or libraries (such as FFmpeg) can be used, but this application example does not limit this.
[0106] For example, the dialogue segment 1 spans from "00:00:00 to 00:00:18". Use the FFm peg command or other video processing tools to extract the video segment between 00:00:00 and 00:00:18 from "Movie A.mp4".
[0107] S204: Select M keyframes from the video clip, where M is a positive integer.
[0108] In this embodiment, M keyframes are selected from a video clip, where M is a positive integer, to generate a film reel. The value of M can be set according to preset requirements (e.g., 20, 30, etc.) or determined based on the video clip. A keyframe is a frame that represents the visual features of a corresponding scene. Keyframes can be selected by extracting frame images from the video file to be segmented at fixed time intervals (e.g., every 0.1 seconds, every 0.2 seconds, every 5 seconds, etc.); or by extracting frame images based on significant changes in the scene content (e.g., scene transitions, significant changes in character actions, etc.); or by determining M keyframes based on the similarity between candidate keyframes and a preset similarity threshold; or by combining a fixed time interval with significant changes in the scene content, first extracting candidate keyframes at fixed time intervals, then extracting frame images from the candidate keyframes based on significant changes in the scene content (e.g., scene transitions, significant changes in character actions, etc.), and finally determining M keyframes based on the similarity between the frame images and a preset similarity threshold. This embodiment does not limit the specific methods described.
[0109] For example, if the preset number of keyframes is 10, one frame can be extracted every 5 seconds, or 10 keyframes can be selected based on similarity calculation.
[0110] Specifically, a video frame extraction tool can be used to extract images from video clips at preset time intervals (such as 0.1s, 0.2s, etc.) as candidate keyframes; similarity calculations (such as the image similarity algorithm SSIM) can be performed on the candidate keyframes to generate a similarity matrix; and based on the similarity matrix, M keyframes that meet the preset similarity threshold can be obtained.
[0111] For example, if the preset requirement is to select 20 keyframes from a video clip, a video frame extraction tool can be used to extract one frame from the video clip every 0.1 seconds (or other intervals) as candidate keyframes. Using an image similarity algorithm (such as SSIM) or other filtering rules, the candidate keyframes are compared pairwise to calculate a similarity matrix. Based on the similarity matrix, frames with a similarity higher than a threshold (such as 0.9) are removed until the number of remaining frames reaches 20. The 20 selected keyframes are saved as JPEG image files, and their corresponding timestamps and video clip information are labeled on each keyframe.
[0112] S205, Generate film images based on M keyframes.
[0113] In this embodiment of the application, a film roll is generated based on M keyframes. The film roll can be an image formed by arranging multiple keyframes together in chronological order or a specific layout, used to visually display important scenes in a video clip. This application example does not limit this.
[0114] For details on how to generate a film roll image from M keyframes, please refer to... Figure 4 The method shown. (As illustrated) Figure 4 The diagram shown illustrates the implementation flow of a film image generation method provided in this application, which may specifically include the following steps:
[0115] S401, sort the M keyframes by time to generate a keyframe sequence.
[0116] In this embodiment of the application, M keyframes are sorted by time to generate a keyframe sequence, such as sorting the keyframes in ascending order according to the time information marked in the keyframes to generate a keyframe sequence.
[0117] If keyframes are not labeled with time information, they can be sorted by extraction order; if there are timestamp conflicts (such as identical time information), they can be sorted by frame content hash value to ensure uniqueness.
[0118] For example, if there are three keyframes with timestamps of 00:00:10, 00:00:20, and 00:00:05, the sorted keyframe sequence would be: keyframe 00:00:05, keyframe 00:00:10, and keyframe 00:00:20.
[0119] S402 performs size normalization on each keyframe in the keyframe sequence.
[0120] In this embodiment, each keyframe in the keyframe sequence is normalized in size. Since the keyframe sizes may be inconsistent, leading to incomplete content in subsequent processing, each keyframe in the keyframe sequence can be normalized to ensure they have the same size. Simultaneously, when normalizing keyframes, it's necessary to ensure they maintain their original proportions and content integrity. This can be achieved through centering or black border padding to ensure the content integrity of the normalized keyframes (e.g., padding the sides with black when the width is insufficient, and padding the top and bottom when the height is insufficient). Image processing tools or libraries (such as OpenCV, Pillow, etc.) can be used to normalize (scale or crop) each keyframe to achieve the target size. The target size can be determined based on the sizes of all keyframes in the keyframe sequence, such as averaging the sizes of all keyframes to obtain the target size, and then adjusting the keyframe sizes to the target size.
[0121] For example, consider three keyframes with original dimensions of 800x600, 1280x720, and 640x480. Averaging the width and height of all keyframes yields a target width of (800+1280+640) / 3 ≈ 906.67 pixels and a target height of (600+720+480) / 3 = 600 pixels. Since the target size is not an integer, it can be rounded to 907x600 pixels. Using the OpenCV library, each keyframe is scaled while maintaining the aspect ratio, and the height is cropped or padded to achieve the target size of 907x600 pixels.
[0122] S403 stitches together all the keyframes in the keyframe sequence after size normalization to generate a film image.
[0123] In this embodiment, all keyframes in the keyframe sequence after size normalization are stitched together to generate a film reel. The keyframes can be stitched together horizontally or vertically in chronological order using image processing tools or libraries (such as OpenCV) to form a continuous "film reel," i.e., the film reel. Furthermore, dividing lines or blank areas can be added between keyframes to improve the readability of the film reel. Additionally, title, timestamp range, and other identifying information are added at the beginning or end of the film reel.
[0124] For example, using the OpenCV library, resized keyframes can be horizontally stitched together, and a title "Video Clip Film Reel" can be added above the stitched image, with a timestamp range "00:00:00-00:00:30" added below, thus generating an image file named "Film Reel.jpeg".
[0125] S206, Determine the node information of dialogue segments in the initial scene segmentation file.
[0126] In this embodiment of the application, the initial scene segmentation file is a file obtained after analyzing and segmenting the video content. It contains detailed information such as the start time, end time, and node information of each scene and dialogue segment in the video. The node information of the dialogue segment is determined in the initial scene segmentation file. The node information can be a structured field in the initial scene segmentation file that identifies the dialogue segment, such as node ID, timestamp, etc. The position of the dialogue segment in the scene segmentation file can be located through the node ID.
[0127] For example: the node ID of dialogue segment 1 is determined to be 1 from the initial scene segmentation file, and the timestamp information is: start time "00:00:10.000" and end time "00:00:20.000".
[0128] S207, Create a corresponding storage node based on the node information of the dialogue segment, and use the storage node as the storage location.
[0129] In this embodiment, a corresponding storage node is created based on the node information of the dialogue segment. The storage node is used as the storage location. The dialogue segment in the initial scene segmentation file can be located based on the node ID of the dialogue segment. A child node (such as image_storage) is added under the dialogue segment node as a storage node, or an associated storage node is created under the scene root node as the storage location of the film image.
[0130] S208. According to the storage location of the film image, store the film image in the initial scene segmentation file to obtain the candidate scene segmentation file.
[0131] In this embodiment, the film images are stored in an initial scene segmentation file according to their storage location to obtain a candidate scene segmentation file. The candidate scene segmentation file can be obtained by storing the path or data of the film images in the initial scene segmentation file according to their corresponding storage location. The candidate scene segmentation file may contain scene time information, dialogue clips, and the film images corresponding to each scene.
[0132] For example, in the "Initial Scene Division.json" file, add a "Film Image Path" field for each scene. For the first scene, save the generated film image as "Scene 1 Film Image.jpg" and store its path " / path / to / Scene 1 Film Image.jpg" in the "Film Image Path" field. Process other scenes in the same way to obtain the candidate scene division file "Candidate Scene Division.json".
[0133] For details on how to save the film images into the initial scene segmentation file according to their storage location to obtain the candidate scene segmentation file, please refer to [example missing]. Figure 5 The method shown. (As illustrated) Figure 5 The diagram shown illustrates the implementation flow of a method for generating candidate scene segmentation files according to an embodiment of this application, which may specifically include the following steps:
[0134] S501, Generate film image link based on film image.
[0135] In this embodiment of the application, after the film image is generated, in order to reference and display the film image in the initial scene segmentation file, a film image link can be generated. The film image link can be a local path of the film image file or a URL of the film image data (i.e., the film image is uploaded to the server to generate a URL).
[0136] For example, you can upload the film image link to the server and obtain the URL L1 of the film image link.
[0137] S502, link the film image to the location where the film image is stored in the initial scene segmentation file, and obtain the candidate scene segmentation file.
[0138] In this embodiment of the application, the film image is linked and stored in the storage location of the film image in the initial scene segmentation file to obtain the candidate scene segmentation file.
[0139] For example, based on the node ID of the dialogue segment, the corresponding storage node is found in the initial scene segmentation file, and the generated film image link is stored in the corresponding attribute of the storage node (such as "image_path"), thus obtaining the candidate scene segmentation file.
[0140] S209: Based on the candidate scene division files and scene segmentation instructions, construct prompt words for the large language model.
[0141] In this embodiment, prompt words for the large language model are constructed based on the candidate scene segmentation file and the scene segmentation description task instructions. The scene segmentation description task may include segmentation criteria (e.g., based on time, content changes, etc.), the number of segments (e.g., how many scenes need to be segmented), and segmentation accuracy requirements. The key information in the candidate scene segmentation file is combined with the scene segmentation description task instructions to generate prompt words for the large language model.
[0142] For example, the prompt words for the constructed large language model could be: "Based on the provided candidate scene segmentation file (containing scene ID, dialogue fragments, timestamps, film reel links, etc.), segment the scenes according to the following segmentation criteria: First, perform preliminary segmentation based on time intervals, then fine-tune according to content changes and visual features to ensure that the number of segmented scenes meets the requirements and the segmentation accuracy reaches the specified standard. Please output the segmented scene information, including scene ID, start time, end time, scene summary, main plot, corresponding film reel link, character name, emotional state, behavioral events, etc."
[0143] S210, input the prompt words into the large language model to obtain structured scene data, which contains G scene information, where G is a positive integer.
[0144] In this embodiment, the prompt words are input into a large language model. The large language model performs scene segmentation based on the information in the prompt words, thereby outputting structured scene data. The structured scene data can be in the format of JSON, XML, etc., and contains G scene information, that is, each scene corresponds to one scene information. The scene information can be scene ID, start time, end time, scene summary, main plot, corresponding film roll link, character name, emotional state, behavioral event, etc. The large language model can read the film roll link in the candidate scene segmentation file according to the prompt words to obtain the corresponding scene image. This embodiment does not limit this.
[0145] S211, Perform scene segmentation on the video file to be segmented based on any scene information.
[0146] In this embodiment of the application, the video file to be segmented is segmented according to any scene information.
[0147] For example, read the structured scene data output by the large language model, locate the corresponding segmentation point in the video file to be segmented based on the start and end times in the structured scene data, use video editing software or tools to segment the video file to be segmented based on the located segmentation point, obtain the segmented scene segments, save the segmented scene segments as independent video files, or perform other forms of processing and storage as required.
[0148] Furthermore, the above describes the process by which a large language model generates structured scene data based on prompt words. After this process, the generated structured scene data can be fed back into the large language model so that the large language model can reflect on the structured scene data and determine whether the word structured scene data is reasonable.
[0149] Corresponding to the above method embodiments, this application also provides a video scene segmentation device, such as... Figure 6 As shown, the device may include an acquisition module 601, a preliminary segmentation module 602, an extraction module 603, a first processing module 604, and a second processing module 605.
[0150] Module 601 retrieves the video file to be segmented and the corresponding subtitle file;
[0151] The initial segmentation module 602 performs scene segmentation on the subtitle file to obtain an initial scene segmentation file. The initial scene segmentation file contains N dialogue segments, each dialogue segment corresponding to a scene, where N is a positive integer.
[0152] Extraction module 603, for any dialogue segment, selects M keyframes from the video file to be segmented based on the dialogue segment and generates a film roll, where M is a positive integer;
[0153] The first processing module 604 determines the storage location of the film image corresponding to the dialogue segment in the initial scene segmentation file, and stores the film image in the initial scene segmentation file according to the storage location of the film image to obtain the candidate scene segmentation file.
[0154] The second processing module 605 performs scene segmentation on the video file to be segmented based on the candidate scene segmentation file.
[0155] This application also provides an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.
[0156] Memory 703 is used to store computer programs;
[0157] In one embodiment of this application, when the processor 701 executes a program stored in the memory 703, it performs the following steps:
[0158] Obtain the video file to be segmented and the corresponding subtitle file. Perform scene segmentation on the subtitle file to obtain an initial scene segmentation file. The initial scene segmentation file contains N dialogue segments, each dialogue segment corresponding to a scene, where N is a positive integer. For any dialogue segment, select M keyframes from the video file to be segmented based on the dialogue segment to generate a film reel, where M is a positive integer. Determine the storage location of the film reel corresponding to the dialogue segment in the initial scene segmentation file. Store the film reel in the initial scene segmentation file according to the storage location of the film reel to obtain a candidate scene segmentation file. Perform scene segmentation on the video file to be segmented based on the candidate scene segmentation file.
[0159] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0160] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0161] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.
[0162] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0163] In another embodiment provided in this application, a storage medium is also provided, which stores instructions that, when run on a computer, cause the computer to execute any of the video scene segmentation methods described in the above embodiments.
[0164] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the video scene segmentation method described in any of the above embodiments.
[0165] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive (SSD)).
[0166] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0167] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0168] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. A method for segmenting a video scene, characterized in that, The method includes: Obtain the video file to be segmented and its corresponding subtitle file; The subtitle file is segmented into scenes to obtain an initial scene segmentation file. The initial scene segmentation file contains N dialogue segments, each dialogue segment corresponding to a scene, where N is a positive integer. For any of the dialogue segments, select M keyframes from the video file to be segmented based on the dialogue segment to generate a film reel, where M is a positive integer; Determine the storage location of the film image corresponding to the dialogue segment in the initial scene segmentation file, and store the film image in the initial scene segmentation file according to the storage location of the film image to obtain the candidate scene segmentation file; Based on the candidate scene segmentation file, the video file to be segmented is segmented into scenes.
2. The method according to claim 1, characterized in that, The process of segmenting the subtitle file into scenes to obtain an initial scene segmentation file includes: Determine the time interval between adjacent texts in the subtitle file; when the time interval is greater than a preset time threshold, set a scene segmentation point between adjacent texts. And / or, Determine the semantic similarity between adjacent texts in the subtitle file. When the semantic similarity is lower than a preset semantic threshold, set scene segmentation points between adjacent texts. The subtitle file is segmented into N dialogue segments based on the scene segmentation points. For any of the dialogue segments, the dialogue segments are analyzed to obtain the corresponding scene summary and dialogue time period; The initial scene segmentation file is generated based on the N dialogue segments, their corresponding scene summaries, and the dialogue time periods.
3. The method according to claim 2, characterized in that, The step of selecting M keyframes from the video file to be segmented based on the dialogue segment and generating a film reel includes: Based on the dialogue time segment corresponding to the dialogue segment, determine the video segment from the video file to be segmented; Select M keyframes from the video segment; The film image is generated based on the M keyframes.
4. The method according to claim 3, characterized in that, The step of generating the film image based on the M keyframes includes: The M keyframes are sorted by time to generate a keyframe sequence; Size normalization is performed on each keyframe in the keyframe sequence; All keyframes in the keyframe sequence after size normalization are stitched together to generate the film image.
5. The method according to claim 1, characterized in that, Determining the storage location of the film reel corresponding to the dialogue segment in the initial scene segmentation file includes: The node information of the dialogue segment is determined in the initial scene segmentation file; Based on the node information of the dialogue segment, a corresponding storage node is created, and the storage node is used as the storage location.
6. The method according to claim 5, characterized in that, The step of storing the film image in the initial scene segmentation file according to the storage location of the film image to obtain the candidate scene segmentation file includes: Generate film image links based on the film images; The film image is linked and stored in the location where the film image is stored in the initial scene segmentation file to obtain the candidate scene segmentation file.
7. The method according to claim 1, characterized in that, The step of segmenting the video file to be segmented according to the candidate scene segmentation file includes: Based on the candidate scene segmentation files and scene segmentation instructions, construct prompt words for the large language model; The prompt words are input into a large language model to obtain structured scene data, which contains G scene information, where G is a positive integer; Based on any of the scene information, the video file to be segmented is segmented into scenes.
8. A video scene segmentation device, characterized in that, The device includes: The acquisition module retrieves the video file to be segmented and its corresponding subtitle file. The initial segmentation module performs scene segmentation on the subtitle file to obtain an initial scene segmentation file. The initial scene segmentation file contains N dialogue segments, each dialogue segment corresponding to a scene, where N is a positive integer. The extraction module, for any one of the dialogue segments, selects M keyframes from the video file to be segmented based on the dialogue segment and generates a film reel, where M is a positive integer; The first processing module determines the storage location of the film image corresponding to the dialogue segment in the initial scene segmentation file, and stores the film image in the initial scene segmentation file according to the storage location of the film image to obtain the candidate scene segmentation file; The second processing module performs scene segmentation on the video file to be segmented based on the candidate scene segmentation file.
9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.