Video clapper method, device, equipment and computer readable storage medium

Abstract

The application provides a video split shot method, device and equipment and a computer readable storage medium. The method comprises: performing frame extraction processing on a target video to obtain a video frame sequence corresponding to the target video; performing feature extraction on each video frame in the video frame sequence to obtain graph embedding features of each video frame; obtaining similarity between graph embedding features of adjacent video frames in the video frame sequence based on the graph embedding features of each video frame; performing split shot processing on the video frame sequence based on the similarity between the graph embedding features of the adjacent video frames to obtain at least two initial split shots, each of which comprises at least one video frame; and performing aggregation processing on the at least two initial split shots to obtain at least two target split shots. Through the application, the split shot accuracy and the split shot aggregation degree can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video splitting method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0003] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and then performs image processing to create images more suitable for human observation or transmission to instruments for detection. Video processing is a crucial technology within computer vision.

[0004] Video scene segmentation is a crucial task in video processing, involving distinguishing which video frames belong to the same shot and depict the same character and event. Scene transition recognition provided by scene segmentation can significantly compress the data extracted from video samples and provide key clues to the changes in the storyline of a video sequence.

[0005] In related technologies, the commonly used method is to process video into segments based on the scale-invariant feature transform (SIFT) features of video frames. However, due to the inherent limitations of SIFT features, the accuracy of segmentation is low, and the aggregation of segments is also low. Summary of the Invention

[0006] This application provides a video scene segmentation method, apparatus, device, and computer-readable storage medium, which can improve the accuracy and cohesion of scene segmentation.

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] This application provides a video scene splitting method, including:

[0009] Frame extraction is performed on the target video to obtain a video frame sequence corresponding to the target video;

[0010] Feature extraction is performed on each video frame in the video frame sequence to obtain the graph embedding features of each video frame;

[0011] Based on the graph embedding features of each video frame, the similarity between the graph embedding features of adjacent video frames in the video frame sequence is obtained.

[0012] Based on the similarity between the graph embedding features of the adjacent video frames, the video frame sequence is processed into a segmentation process to obtain at least two target segments, each of which includes at least one video frame.

[0013] This application provides a video split-screen device, including:

[0014] The frame extraction module is used to extract frames from the target video to obtain a video frame sequence corresponding to the target video.

[0015] The extraction module is used to extract features from each video frame in the video frame sequence to obtain the graph embedding features of each video frame;

[0016] The acquisition module is used to acquire the similarity between the graph embedding features of adjacent video frames in the video frame sequence based on the graph embedding features of each video frame.

[0017] The storyboard module is used to perform storyboard processing on the video frame sequence based on the similarity between the graph embedding features of the adjacent video frames to obtain at least two initial storyboards, each of the initial storyboards including at least one video frame.

[0018] The aggregation module is used to aggregate at least two initial storyboards to obtain at least two target storyboards.

[0019] In the above scheme, the frame extraction module is also used to obtain the video frame rate and the duration of the target video;

[0020] The number of frames to be extracted is determined based on the target video's frame rate and duration.

[0021] Based on the stated number of frames extracted, the target video is subjected to frame extraction processing to obtain a video frame sequence containing the stated number of video frames extracted.

[0022] In the above scheme, the extraction module is further used to extract features from each video frame in the video frame sequence using a first graph embedding model to obtain graph embedding features of each video frame.

[0023] The device further includes:

[0024] The update module is used to filter out difficult storyboard examples from at least two target storyboards;

[0025] Obtain the manual storyboard result corresponding to the storyboard difficulty example, and the manual storyboard result is used to indicate the video frames belonging to the same storyboard;

[0026] A training set is constructed based on the difficult storyboard examples and the manual storyboard results. The model parameters of the first image embedding model are updated using the training set to obtain the second image embedding model.

[0027] In the above scheme, the update module is also used to obtain the number of video frames in each target segment and the average number of video frames corresponding to each target segment;

[0028] Determine the ratio of the number of video frames in each target scene to the average number of video frames;

[0029] A storyboard where the ratio of the number of video frames to the average number of video frames does not reach the target ratio is considered a difficult storyboard example.

[0030] In the above scheme, the update module is further configured to perform the following operations for each target scene when the target scene includes at least two video frames:

[0031] Starting from the first video frame in the target storyboard, video frames of the target proportion are obtained according to the playback time sequence to obtain the first subset of video frames, and

[0032] Starting from the last video frame in the target segment, video frames of the target proportion are obtained in reverse order according to the playback time points to obtain the second subset of video frames.

[0033] Obtain the similarity between the first subset of video frames and the second subset of video frames;

[0034] When the similarity does not reach the similarity threshold, the target storyboard is determined to be a difficult storyboard example.

[0035] In the above scheme, the training set includes multiple triples, and each triple includes a reference image, a positive image belonging to the same shot as the reference image, and a negative image belonging to a different shot than the reference image;

[0036] The update module is also used to input the reference image, positive image, and negative image in the triplet into the first image embedding model;

[0037] The first graph embedding model is used to perform forward computation on the reference image, positive image, and negative image to predict the graph embedding features of the reference image, the positive image, and the negative image.

[0038] The first difference between the graph embedding features of the reference image and the graph embedding features of the positive image, and the second difference between the graph embedding features of the reference image and the graph embedding features of the negative image are obtained, and the value of the loss function of the first graph embedding model is determined based on the first difference and the second difference.

[0039] Based on the value of the loss function, the model parameters of the first graph embedding model are updated.

[0040] In the above scheme, the update module is further used to obtain a test set and input the test set into the first graph embedding model and the second graph embedding model respectively to obtain a first prediction result corresponding to the first graph embedding model and a second prediction result corresponding to the second graph embedding model;

[0041] Based on the first prediction result, determine the value of the loss function corresponding to the first graph embedding model;

[0042] Based on the second prediction result, determine the value of the loss function corresponding to the second graph embedding model;

[0043] When the value of the loss function corresponding to the second graph embedding model is less than the value of the loss function corresponding to the first graph embedding model, the first graph embedding model is updated using the second graph embedding model.

[0044] In the above scheme, the storyboard module is further used to obtain a first similarity threshold and a second similarity threshold, wherein the first similarity threshold is greater than the second similarity threshold;

[0045] Two video frames whose similarity between graph embedding features reaches the first similarity threshold are grouped into the same shot.

[0046] And two video frames whose similarity between graph embedding features does not reach the second similarity threshold are divided into different sub-shots to obtain at least two initial sub-shots.

[0047] In the above scheme, the storyboard module is also used to obtain at least two triplets, wherein the triplets include a reference image, a positive image belonging to the same storyboard as the reference image, and a negative image belonging to a different storyboard as the reference image;

[0048] Obtain at least two candidate similarity thresholds;

[0049] Based on the triples, determine the storyboard accuracy and storyboard balance score corresponding to each candidate similarity.

[0050] The candidate similarity threshold with the highest storyboard balance score is used as the first similarity threshold;

[0051] The candidate similarity threshold with the highest storyboard accuracy is used as the second similarity threshold.

[0052] In the above scheme, the aggregation module is also used to obtain the similarity between the initial storyboards;

[0053] Based on the similarity between the initial storyboards, the at least two initial storyboards are aggregated so that the similarity between any two initial storyboards aggregated into the same target storyboard reaches a similarity threshold.

[0054] In the above scheme, the aggregation module is further configured to, when the initial storyboard includes at least two video frames, use the first initial storyboard and the second initial storyboard as any two of the at least two initial storyboards, and perform the following processing on any two initial storyboards:

[0055] For any target video frame in the first initial storyboard, the similarity between the target video frame and each video frame in the second initial storyboard is obtained, and at least two similarities are obtained for the target video frame.

[0056] Based on at least two similarities corresponding to each target video frame in the first initial segment, determine the maximum similarity corresponding to each target video frame in the first initial segment;

[0057] Obtain the first number of target video frames whose maximum similarity reaches the first similarity threshold, and the second number of target video frames in the first initial storyboard;

[0058] The ratio of the first quantity to the second quantity is used as the similarity between the first initial storyboard and the second initial storyboard.

[0059] In the above scheme, the aggregation module is further configured to, when the initial storyboard includes at least two video frames, use the first initial storyboard and the second initial storyboard as any two of the at least two initial storyboards, and perform the following processing on any two initial storyboards:

[0060] When the playback time sequence of the first initial scene is earlier than that of the second initial scene, the target video frame with the last playback time sequence in the first initial scene is obtained.

[0061] Obtain the similarity between the target video frame and each video frame in the second initial storyboard, and obtain at least two similarities corresponding to the target video frame;

[0062] The average similarity of at least two similarities is used as the similarity between the first initial storyboard and the second initial storyboard.

[0063] This application provides a computer device, including:

[0064] Memory, used to store executable instructions;

[0065] The processor, when executing executable instructions stored in the memory, implements the video splitting method provided in the embodiments of this application.

[0066] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the video splitting method provided in this application.

[0067] The embodiments of this application have the following beneficial effects:

[0068] Applying the above embodiments, a video frame sequence corresponding to the target video is obtained by performing frame extraction processing on the target video; feature extraction is performed on each video frame in the video frame sequence to obtain the graph embedding features of each video frame; based on the graph embedding features of each video frame, the similarity between the graph embedding features of adjacent video frames in the video frame sequence is obtained; based on the similarity between the graph embedding features of adjacent video frames, the video frame sequence is processed into a storyboard to obtain at least two initial storyboards, each of which includes at least one video frame; compared with SIFT features, the graph embedding features of this application can more accurately describe video frames, thereby improving the accuracy of storyboarding; furthermore, this application improves the aggregation degree of storyboarding by aggregating the initial storyboards, resulting in a higher degree of internal aggregation within the target storyboard. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the result of manually segmenting a video.

[0070] Figure 2 This is a schematic diagram of the video segmentation result based on SIFT features;

[0071] Figure 3 This is a schematic diagram of an optional architecture of the video storyboard system provided in the embodiments of this application;

[0072] Figure 4 This is a flowchart illustrating the video scene splitting method provided in an embodiment of this application;

[0073] Figure 5 This is a schematic diagram of the ResNet module provided in an embodiment of this application;

[0074] Figure 6 This is a schematic diagram of the process for determining the initial storyboard provided in an embodiment of this application;

[0075] Figure 7 This is a schematic diagram of the process for determining the initial storyboard provided in an embodiment of this application;

[0076] Figure 8 This is a schematic diagram illustrating the aggregation effect provided in the embodiments of this application;

[0077] Figure 9 This is a video scene splitting method provided in the embodiments of this application;

[0078] Figure 10This is a flowchart illustrating the video scene splitting method provided in an embodiment of this application;

[0079] Figure 11 This is a schematic diagram of the video scene processing flow provided in the embodiments of this application;

[0080] Figure 12 This is a schematic diagram of the video splitting device provided in the embodiments of this application;

[0081] Figure 13 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0083] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0084] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0085] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0086] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0087] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0088] 1) Video scene segmentation refers to the use of different positions and angles in film and television scenes. Each change of camera will cause the video scene to change, while the images in the same scene are the same (often the same person and the same background environment).

[0089] One related technology provides a video scene segmentation method, which is implemented based on the SIFT video switching tool (pyScenedetect) library. That is, the SIFT video switching tool (pyScenedetect) library is used to extract SIFT features from each video frame in the video to be segmented, and then the scene segmentation is performed based on the similarity of SIFT features of adjacent video frames in the video to be segmented.

[0090] Figure 1 This is a schematic diagram illustrating the result of manually segmenting a video. Figure 2 This is a schematic diagram illustrating the result of video segmentation based on SIFT features. (See attached image) Figure 1 and Figure 2 In contrast Figure 1 , Figure 2 The storyboard results in the middle are missing a storyboard 201.

[0091] The applicant discovered the following problems in the above solution when implementing the embodiments of this application:

[0092] 1) Due to the inherent limitations of SIFT's expressive power, it cannot be applied to distinguish all images in massive datasets, such as different athletes wearing the same sportswear.

[0093] 2) In image noise distortion or deformation, corner features can easily cause errors in image representation, so that seemingly similar images cannot be matched to the same scene under this feature representation;

[0094] 3) It is difficult to distinguish fine-grained differences. For example, since the corner points of a face in SIFT features are often located at the same positions as the eyes and nose, but it does not have the ability to specifically distinguish the eyes and noses of different people, it cannot distinguish different facial structures when the face scene changes.

[0095] 4) Since it is a logic-driven storyboard framework rather than a data-driven storyboard framework, the “adjacent feature similarity storyboard” is not applicable to other features, and SIFT features do not have feature learning capabilities and are difficult to optimize.

[0096] Based on this, embodiments of this application provide a video scene splitting method, apparatus, device, and computer-readable storage medium, which can solve at least one of the above problems.

[0097] See Figure 3 , Figure 3This is an optional architecture diagram of the video storyboard system provided in this application embodiment. To support an exemplary application, the terminal (terminal 400-1 and terminal 400-2 are shown as examples) connects to the server 200 via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. In practical applications, the terminal is equipped with a client, such as a video client, a browser client, a news client, an educational client, etc., and the number of terminals and servers is unlimited.

[0098] The terminal is used to receive the input target video and send the target video to the server;

[0099] Server 200 is used to perform frame extraction processing on the target video to obtain a video frame sequence corresponding to the target video; to extract features from each video frame in the video frame sequence to obtain graph embedding features of each video frame; to obtain the similarity between graph embedding features of adjacent video frames in the video frame sequence based on the graph embedding features of each video frame; to perform scene segmentation processing on the video frame sequence based on the similarity between graph embedding features of adjacent video frames to obtain at least two initial scenes, each initial scene including at least one video frame; and to perform aggregation processing on the at least two initial scenes to obtain at least two target scenes.

[0100] The terminal is used to acquire at least two target storyboards.

[0101] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart TV, etc., but is not limited to these.

[0102] Based on the above description of the video scene splitting system of the embodiments of this application, the video scene splitting method provided by the embodiments of this application will be described below. See also Figure 4 , Figure 4 This is a flowchart illustrating the video scene splitting method provided in this application embodiment. In some embodiments, the video scene splitting method can be implemented by the terminal alone, by the server alone, or by the terminal and the server collaboratively. Taking the server alone as an example, the video scene splitting method provided in this application embodiment includes:

[0103] Step 401: The server performs frame extraction on the target video to obtain the corresponding video frame sequence.

[0104] The source and type of the target video in this application embodiment are not limited. The server can obtain the target video from online business flow, database or other data sources. The target video can be a video published by a video creator, a chat video in instant messaging, or an advertising video published by e-commerce.

[0105] In actual implementation, after the server obtains the target video, it obtains the time points at which frames are to be extracted. This can be done by uniformly extracting frames over time, such as setting a time point every second, that is, extracting one frame every second. Alternatively, it can extract video frames at specified time points. These specified time points can be uniform or non-uniform, such as 1 second, 2 seconds, or 5 seconds.

[0106] In some embodiments, the target video can be processed by the following frame extraction process to obtain a video frame sequence of the target video: obtaining the video frame rate and duration of the target video; determining the number of frames to be extracted based on the video frame rate and duration of the target video; and performing frame extraction on the target video based on the number of frames to obtain a video frame sequence containing the number of extracted video frames.

[0107] Here, video frame rate is a measure of the number of frames displayed, i.e., the number of frames displayed per second. In practice, the frame extraction time points can be determined based on the video frame rate. That is, for any given second, the same number of time points as the video frame rate are determined. For example, if the video frame rate is 25, then 25 video frames need to be extracted every second, or one video frame is extracted every 0.04 seconds. Based on the target video's video frame rate and duration, the number of frames to be extracted can be determined. For example, if the video frame rate is 25 and the duration is 4 seconds, then a total of 100 video frames are extracted. These video frames are arranged into a video frame sequence according to their playback time points.

[0108] Thus, by obtaining the video frame rate and duration of the target video, and determining the number of frames to be extracted based on the video frame rate and duration of the target video, this application can make the extracted video frames more coherent and facilitate subsequent storyboarding.

[0109] Step 402: Extract features from each video frame in the video frame sequence to obtain the graph embedding features of each video frame.

[0110] In some embodiments, a first graph embedding model can be used to extract features from each video frame in the video frame sequence to obtain the graph embedding features of each video frame.

[0111] Here, the first image embedding model is trained on the training set. In actual implementation, the server constructs a neural network model, such as a convolutional neural network model, and takes the training set containing multiple image samples as input to construct the neural network model. The loss function is calculated based on the output, and then the loss function value is backpropagated to each layer of the neural network model. The model parameters of each layer are updated using stochastic gradient descent (SGD) to train the neural network model, resulting in the first image embedding model. After obtaining the first image embedding model, feature extraction is performed on video frames in the video frame sequence using the trained first image embedding model.

[0112] This application uses a first graph embedding model to extract features from each video frame in a video frame sequence, obtaining graph embedding features for each video frame. Since the first graph embedding model of this application is trained on a training set containing multiple image samples, the graph embedding features extracted by the first graph embedding model can better represent video frames, thereby improving the accuracy of scene segmentation.

[0113] In some embodiments, the first graph embedding model may use ResNet101 as the network structure. See [link to documentation]. Figure 5 , Figure 5 This is a schematic diagram of the ResNet module provided in the embodiment of this application. Here, a 1×1 convolution is used to reduce the 256 dimensions of the input to 64 dimensions. Then, after a 3×3 convolution, a 1×1 convolution is used to increase the dimensionality back to 256 dimensions, which can reduce the amount of parameter computation.

[0114] Here, ResNet101 includes a feature module and an embedding module. Table 1 is the feature module structure table of ResNet101 provided in the embodiments of this application. Referring to Table 1, the feature module of ResNet101 provided in the embodiments of this application includes 5 convolutional layers, namely Conv1, Conv2_x, Conv3_x, Conv4_x, and Conv5_x. Among them, Conv1 is a 7×7×64 convolution with a stride of 2. Conv2_x includes a 3×3 max pooling layer and 3 ResNet blocks. Conv3_x-Conv5_x include 3 ResNet blocks, 4 ResNet blocks, 23 ResNet blocks, and 3 ResNet blocks, respectively. Table 2 is the embedding module structure table of ResNet101 provided in the embodiments of this application. Referring to Table 2, the embedding module includes a max pooling layer.

[0115]

[0116] Table 1. Feature module structure of ResNet101

[0117]

[0118] Table 2. Structure of the ResNet101 Embedding Module

[0119] In practice, during training, a classification layer (i.e., a fully connected layer) can be added after the max pooling layer. This classification layer uses the graph embedding features output by the max pooling layer to classify and predict the images, resulting in a list of images. In practical applications, images labeled with image categories are used as training samples. These training samples are input into the ResNet101 network structure, which outputs the graph embedding features corresponding to the training samples. These graph embedding features are then input into the fully connected layer, which outputs the image categories of the training samples. Based on the difference between the output image categories and the labeled image categories, the value of the loss function is determined. Finally, based on the value of the loss function, the model parameters of each layer are updated.

[0120] Step 403: Based on the graph embedding features of each video frame, obtain the similarity between the graph embedding features of adjacent video frames in the video frame sequence.

[0121] Here, similarity can be calculated based on Minkowski distance, Euclidean distance, cosine similarity, Pearson correlation coefficient, etc.

[0122] For example, taking cosine similarity as an example, graph embedding features can be represented in the form of vectors. For any two adjacent video frames in a video frame sequence, the graph embedding features of these two video frames are obtained, and the cosine of the angle between these two graph embedding features is calculated. The calculated cosine of the angle is used as the similarity between the two graph embedding features.

[0123] Step 404: Based on the similarity between the graph embedding features of adjacent video frames, perform scene segmentation on the video frame sequence to obtain at least two initial scenes.

[0124] Each initial storyboard includes at least one video frame.

[0125] In practice, it is determined whether the similarity between the graph embedding features of adjacent video frames meets the similarity condition. If it does, it is determined that the adjacent video frames belong to the same scene, so that the video frame sequence can be processed into scenes. Here, the video frames in the same initial scene are continuous in playback time.

[0126] In some embodiments, a similarity threshold is obtained; two video frames whose similarity between graph embedding features reaches the similarity threshold are divided into the same shot to obtain at least two initial shots.

[0127] In actual implementation, for any two adjacent video frames, where the two adjacent video frames are the first video frame and the second video frame, the playback time point of the first video frame is compared with the playback time point of the second video frame. It is then determined whether the similarity between the graph embedding features of the first video frame and the second video frame reaches a first similarity threshold. If so, the second video frame is assigned to the same initial shot as the first video frame; otherwise, a new initial shot is created and the second video frame is assigned to the new initial shot.

[0128] Figure 6 This is a schematic diagram of the initial storyboard determination process provided in the embodiments of this application. See also: Figure 6 If the similarity between the graph embedding features of the first and second video frames reaches the similarity threshold, the first and second video frames are assigned to initial scene 1. If the similarity between the graph embedding features of the second and third video frames reaches the similarity threshold, the third video frame is assigned to the same initial scene as the second video frame, i.e., initial scene 1. If the similarity between the graph embedding features of the third and fourth video frames does not reach the similarity threshold, the fourth video frame is assigned to a different initial scene than the third video frame, i.e., initial scene 2. If the similarity between the graph embedding features of the fifth and fourth video frames reaches the similarity threshold, the fifth video frame is assigned to the same initial scene as the fourth video frame, i.e., initial scene 2.

[0129] In some embodiments, a first similarity threshold and a second similarity threshold are obtained, wherein the first similarity threshold is greater than the second similarity threshold; two video frames whose similarity between graph embedding features reaches the first similarity threshold are assigned to the same scene, and two video frames whose similarity between graph embedding features does not reach the second similarity threshold are assigned to different scenes, so as to obtain at least two initial scenes.

[0130] In actual implementation, for any two adjacent video frames (here, the first video frame and the second video frame), the playback time of the first video frame is compared with the playback time of the second video frame. It is then determined whether the similarity between the graph embedding features of the first and second video frames reaches a first similarity threshold. If so, the second video frame is assigned to the same initial shot as the first video frame. Otherwise, it is determined whether the similarity between the graph embedding features of the first and second video frames reaches a second similarity threshold. If so, the second video frame is considered a transition frame and is not assigned to any initial shot. Otherwise, a new initial shot is created and the second video frame is assigned to the new initial shot.

[0131] For example, Figure 7 This is a schematic diagram of the initial storyboard determination process provided in the embodiments of this application. See also: Figure 7If the similarity between the graph embedding features of the first video frame and the second video frame reaches the first similarity threshold, the first video frame and the second video frame are assigned to initial scene 1. If the similarity between the graph embedding features of the second video frame and the third video frame reaches the first similarity threshold, the third video frame is assigned to the same initial scene as the second video frame, i.e., initial scene 1. If the similarity between the graph embedding features of the third video frame and the fourth video frame does not reach the first similarity threshold, but the similarity between the graph embedding features of the third video frame and the fourth video frame reaches the second similarity threshold, the fourth video frame is not assigned to any initial scene. If the similarity between the graph embedding features of the fifth video frame and the fourth video frame does not reach the first similarity threshold and does not reach the second similarity threshold, the fifth video frame is assigned to an initial scene different from the fourth video frame, i.e., initial scene 2.

[0132] In practical applications, there are transitional video frames in video frame sequences, that is, video frames used to enhance visual effects during shot transitions, such as fade-in / fade-out, image overlap, zoom in / out, etc. This application does not consider these video frames when performing shot segmentation. Based on this, when performing shot segmentation, two video frames whose similarity between image embedding features reaches the first similarity threshold are assigned to the same shot, and two video frames whose similarity between image embedding features does not reach the second similarity threshold are assigned to different shots, while video frames located between the first and second similarity thresholds are not considered. In this way, the influence of transitional video frames on the overall shot segmentation result can be avoided, and the accuracy of shot segmentation can be improved.

[0133] In some embodiments, a first similarity threshold and a second similarity threshold can be obtained by: obtaining at least two triplets, each triplet including a reference image, a positive image belonging to the same scene as the reference image, and a negative image belonging to a different scene from the reference image; obtaining at least two candidate similarity thresholds; determining the scene accuracy and scene balance score corresponding to each candidate similarity threshold based on the triplets; using the candidate similarity threshold with the highest scene balance score as the first similarity threshold; and using the candidate similarity threshold with the highest scene accuracy as the second similarity threshold.

[0134] In actual implementation, multiple candidate similarity thresholds are preset. For example, a candidate similarity threshold is set every 0.01 from 0.05 to 0.99. Then, the multiple candidate similarity thresholds are traversed. For each candidate similarity threshold, the graph embedding features of the reference image, positive image, and negative image in the triplet are obtained. The similarity between the graph embedding features of the reference image and the positive image, and the similarity between the graph embedding features of the reference image and the positive image are obtained. The number of triplets in the multiple triplets where the similarity between the graph embedding features of the reference image and the positive image is greater than the candidate similarity threshold, and the similarity between the graph embedding features of the reference image and the negative image is less than the candidate similarity threshold, is obtained. The ratio of this number to the total number of triplets is used as the storyboard accuracy. The system obtains a first number of triplets in which the similarity between the graph embedding features of the reference image and the positive image is greater than the candidate similarity threshold, and a second number of triplets in which the similarity between the graph embedding features of the reference image and the negative image is greater than the candidate similarity threshold. The sum of the first number and the second number is determined, and the ratio of the first number to the sum of the first number and the second number is used as the recall rate.

[0135] After obtaining the segmentation accuracy and segmentation recall based on the similarity thresholds for each candidate, we can then... Obtain the storyboard balance score, where precision is the accuracy and recall is the recall. Here, based on the obtained storyboard accuracy and storyboard balance score, the candidate similarity threshold with the highest balance score is used as the first similarity threshold; the candidate similarity threshold with the highest accuracy is used as the second similarity threshold.

[0136] In practical applications, if the second similarity threshold obtained based on the above method is greater than the first similarity threshold, the second similarity threshold can be determined according to thr2=thr1-0.05, where thr1 is the first similarity threshold and thr2 is the second similarity threshold.

[0137] Since the first similarity threshold and the second similarity threshold of this application are obtained by filtering the graph embedding features of each image in the triplet, they will be modified accordingly depending on the different graph embedding models. Compared with the fixed first similarity threshold and the second similarity threshold, the first similarity threshold and the second similarity threshold determined in this application can make the storyboard more accurate.

[0138] Step 405: Aggregate at least two initial storyboards to obtain at least two target storyboards.

[0139] In practice, the content of the same storyboard can be played at different times. For example, if a video includes a dialogue between two people, there will be back-and-forth switching between the two shots. Based on this, this application performs aggregation processing on at least two initial storyboards to aggregate the footage shot by the same shot into a single target storyboard, thereby improving the aggregation degree of the storyboards.

[0140] For example, Figure 8 This is a schematic diagram illustrating the aggregation effect provided in the embodiments of this application. See also: Figure 8 , Figure 8 The initial storyboards 801A and 801B were shot from the same camera. Although other storyboards were inserted in between, they can be combined into a single target storyboard. Similarly, the initial storyboards 802A and 802B were shot from the same camera. The initial storyboards 802A and 802B can be combined into a single target storyboard.

[0141] In some embodiments, at least two initial scenes can be aggregated to obtain at least two target scenes by: obtaining the similarity between the initial scenes; and based on the similarity between the initial scenes, aggregating at least two initial scenes so that the similarity between any two initial scenes aggregated into the same target scene reaches a similarity threshold.

[0142] Here, the number of initial scenes aggregated into the same target scene can be two or more. In actual implementation, for each initial scene, the similarity between the initial scene and each other initial scene is obtained. If the similarity between the initial scene and another initial scene reaches a similarity threshold, then the two initial scenes are aggregated together to obtain the target scene. If other initial scenes need to be aggregated into the target scene, then the initial scene needs to reach a similarity threshold with each initial scene in the target scene.

[0143] For example, if the initial storyboards A, B, and C are merged into a single target storyboard, then the similarity between storyboard A and storyboard B reaches a similarity threshold, the similarity between storyboard A and storyboard C reaches a similarity threshold, and the similarity between storyboard B and storyboard C reaches a similarity threshold.

[0144] In some embodiments, the similarity between initial storyboards can be obtained in the following manner: when the initial storyboard includes at least two video frames, the first initial storyboard and the second initial storyboard are taken as any two initial storyboards among the at least two initial storyboards, and the following processing is performed on any two initial storyboards: for any target video frame in the first initial storyboard, the similarity between the target video frame and each video frame in the second initial storyboard is obtained respectively, to obtain at least two similarities corresponding to the target video frame; based on the obtained at least two similarities corresponding to each target video frame in the first initial storyboard, the maximum similarity corresponding to each target video frame in the first initial storyboard is determined; the first number of target video frames whose maximum similarity reaches a third similarity threshold and the second number of target video frames in the first initial storyboard are obtained; the ratio of the first number to the second number is taken as the similarity between the first initial storyboard and the second initial storyboard.

[0145] In actual implementation, if the first initial storyboard includes m video frames and the second initial storyboard includes n video frames, then for any target video frame in the first initial storyboard, the similarity between the target video frame and the n video frames in the second initial storyboard is obtained, resulting in n similarity scores. The maximum similarity among these n similarity scores is then taken as the maximum similarity score corresponding to the target video frame. Therefore, for the m video frames in the first initial storyboard, m maximum similarity scores can be obtained. These m maximum similarity scores are compared with a third similarity threshold, and the first number of these m maximum similarity scores that reach the third similarity threshold is obtained. Then, the ratio of the first number to m is taken as the similarity score between the first initial storyboard and the second initial storyboard.

[0146] Here, when determining similarity, the graph embedding features of each video frame are obtained, and then the similarity between the target video frame and the n video frames in the second initial segment is determined by obtaining the similarity between the graph embedding features of the target video frame and the n video frames in the second initial segment.

[0147] For example, if the first initial storyboard includes 5 video frames A, B, C, D, and E, and the second initial storyboard includes 4 video frames F, G, H, and J, then the similarity between the graph embedding features of A and the graph embedding features of F, G, H, and J is obtained, and the highest similarity is selected as the similarity corresponding to A. For example, if the similarities are 0.8, 0.75, 0.82, and 0.71, then 0.82 is taken as the similarity corresponding to A. The same method is used to obtain the similarities corresponding to B, C, D, and E. For example, if the obtained similarities for A, B, C, D, and E are 0.82, 0.73, 0.78, 0.77, and 0.74 respectively, and the third similarity threshold is 0.75, then the similarity between the first and second initial storyboards is determined to be 0.6.

[0148] In some embodiments, the similarity between initial storyboards can be obtained in the following way: when the initial storyboard includes at least two video frames, the first initial storyboard and the second initial storyboard are taken as any two initial storyboards among the at least two initial storyboards, and the following processing is performed on any two initial storyboards: when the playback time order of the first initial storyboard is earlier than that of the second initial storyboard, the target video frame with the last playback time point in the first initial storyboard is obtained; the similarity between the target video frame and each video frame in the second initial storyboard is obtained to obtain at least two similarities corresponding to the target video frame; the average similarity of the at least two similarities is taken as the similarity between the first initial storyboard and the second initial storyboard.

[0149] In practice, video frames in the same storyboard should typically be continuous in playback time. Based on this, the last target video frame of the initial storyboard that plays earlier can be obtained. Then, the target video frame is matched with each video frame in the initial storyboard that plays later to obtain the similarity between the target video frame and each video frame. Finally, the average of these similarities is obtained to obtain the similarity between the first and second initial storyboards.

[0150] For example, the first initial storyboard includes 5 video frames A, B, C, D, and E (sorted by playback time), and the second initial storyboard includes 4 video frames F, G, H, and J. Then, the similarity between the graph embedding features of A and the graph embedding features of F, G, H, and J is obtained, such as 0.8, 0.75, 0.82, and 0.71. Then, the average of these similarities is obtained, such as 0.77, where the average of 0.8, 0.75, 0.82, and 0.71 is 0.77. Therefore, 0.77 is taken as the similarity between the first and second initial storyboards.

[0151] In some embodiments, after aggregating at least two initial storyboards to obtain at least two target storyboards, the server can further filter out difficult storyboard examples from the at least two target storyboards; obtain the manual storyboard results for the corresponding difficult storyboard examples, which are used to indicate video frames belonging to the same storyboard; construct a training set based on the difficult storyboard examples and the manual storyboard results, and update the model parameters of the first graph embedding model using the training set to obtain the second graph embedding model.

[0152] In practice, after obtaining the target storyboard, difficult storyboard examples can be selected from at least two target storyboards. These difficult storyboard examples are those judged as having storyboard errors based on difficulty judgment conditions, such as the number of video frames in the target storyboard not reaching a certain threshold. Then, the selected difficult storyboard examples are manually judged, i.e., whether the video frames in the difficult storyboard example belong to the same storyboard. If the video frames in the target storyboard are not judged to belong to the same storyboard, the video frames in the target storyboard are processed into storyboards to obtain the manual storyboard result. Here, a training set is constructed based on the difficult storyboard examples and the manual storyboard result. The training set is input into the first graph embedding model, and the model parameters of the first graph embedding model are updated to obtain the second graph embedding model.

[0153] This application constructs a training set based on difficult storyboard examples and manual storyboard results, and updates the model parameters of the first image embedding model using the training set to obtain the second image embedding model. In this way, the first image embedding model can be updated according to the storyboard results, thereby improving the feature expression ability of the first image embedding model and thus improving the accuracy of subsequent storyboards.

[0154] In practice, after obtaining the target storyboard, it can be determined whether each target storyboard is incorrect. If so, it is identified as a difficult storyboard example. In some embodiments, difficult storyboard examples can be selected from at least two target storyboards in the following way: obtain the number of video frames in each target storyboard and the average number of video frames corresponding to each target storyboard; determine the ratio of the number of video frames in each target storyboard to the average number; and identify the target storyboards whose ratio of the number of video frames to the average number does not reach the target ratio as difficult storyboard examples.

[0155] In actual implementation, the total number of video frames in all target scenes is obtained, and then the ratio of the total number of video frames to the number of target scenes is determined as the average number of video frames corresponding to the target scene. For each target scene, the number of video frames in the target scene is obtained, and the ratio of the number of video frames to the average number of video frames is determined. If it is less than the target ratio, the target scene is determined to be a difficult scene example.

[0156] For example, the target ratio can be set to 1 / 4. The number of video frames in all target scenes can be obtained. If there are five target scenes with the numbers 8, 12, 13, 2, and 10 respectively, then the average number of video frames in at least two target scenes corresponds to 9 per target scene. Therefore, the ratio of the number of video frames in the fourth target scene to the average number is less than 1 / 4. Thus, this target scene is a difficult example of a scene.

[0157] In some embodiments, the number of video frames in the target segment can be used to determine whether the target segment is a difficult segment, that is, to determine whether the number of video frames in the target segment reaches a certain threshold. If it does not reach the threshold, the target segment is considered a difficult segment. For example, a target segment with less than 2 video frames can be identified as a difficult segment.

[0158] In some embodiments, difficult scenarios can be selected from at least two target scenarios in the following manner: when a target scenario includes at least two video frames, the following operations are performed for each target scenario: starting from the first video frame in the target scenario, video frames of the target proportion are obtained in the order of playback time points to obtain a first subset of video frames, and starting from the last video frame in the target scenario, video frames of the target proportion are obtained in reverse order of playback time points to obtain a second subset of video frames; the similarity between the first subset of video frames and the second subset of video frames is obtained; when the similarity does not reach a fourth similarity threshold, the target scenario is determined to be a difficult scenario.

[0159] In the course of implementing this application, the applicant discovered that common difficulties in storyboarding stem from " To address the issue of incorrect storyboards, this application determines storyboard errors by comparing video frames with the proportions of the preceding and following targets within the target storyboard, thereby ensuring the accuracy of storyboard error determination for intermediate transition video frames.

[0160] In practice, when calculating the similarity between the first and second video frame subsets, the same method as used above for determining the similarity between the initial storyboards can be used to calculate the similarity between the first and second video frame subsets.

[0161] For example, if the target ratio is set to 1 / 3, then according to the playback timeline, the first 1 / 3 of the video frames and the last 1 / 3 of the video frames in the target storyboard are obtained. The first 1 / 3 of the video frames are taken as the first subset of video frames, and the last 1 / 3 of the video frames are taken as the second subset of video frames. For each target video frame in the first subset of video frames, the similarity between the target video frame and each video frame in the second subset of video frames is obtained. From the multiple similarities obtained, the maximum similarity is taken as the similarity corresponding to the target video frame. The similarity corresponding to all target video frames in the first subset of video frames is obtained, and the similarity corresponding to each target video frame is compared with the similarity threshold to determine the number of similarities that reach the third similarity threshold. The ratio of the number of similarities to the number of video frames in the first subset of video frames is determined as the similarity between the first subset of video frames and the second subset of video frames. When the similarity does not reach the fourth similarity threshold, the target storyboard is determined to be a difficult storyboard example.

[0162] In some embodiments, the training set includes multiple triples, each triple including a reference image, a positive image belonging to the same shot as the reference image, and a negative image belonging to a different shot than the reference image. The model parameters of the first graph embedding model can be updated based on the value of the loss function in the following manner: the reference image, positive image, and negative image in the triple are input into the first graph embedding model; the first graph embedding model performs forward computation on the reference image, positive image, and negative image to predict the graph embedding features of the reference image, the positive image, and the negative image; a first difference between the graph embedding features of the reference image and the graph embedding features of the positive image, and a second difference between the graph embedding features of the reference image and the graph embedding features of the negative image are obtained, and the value of the loss function of the first graph embedding model is determined based on the first difference and the second difference; the model parameters of the first graph embedding model are updated based on the value of the loss function.

[0163] In practice, multiple difficult case pairs A and B can be determined based on the difficult case examples and the results of manual case analysis. Here, A and B each represent a case, the video frames in A are similar to each other, the video frames in B are similar to each other, and the video frames in A and B are dissimilar to each other. That is, the video frames within the set are similar to each other, while the video frames outside the set are dissimilar to each other. Multiple triples are extracted from the difficult case pairs. For example, A is {a1,a2,…}, B is {b1,b2,b3,…}, and (a1, a2, b1), (a1, a3, b2), etc. can be extracted from A and B. Taking (a1, a2, b1) as an example, a1 is the reference image, a2 and a1 belong to the same case, and b1 and a1 belong to different case. That is, a2 is the positive image, and b1 is the negative image.

[0164] After obtaining the triples, the images in the triples are input into the first graph embedding model. The first graph embedding model performs forward computation on the reference image, the positive image, and the negative image to predict the graph embedding features of the reference image, the positive image, and the negative image. Here, when updating the first graph embedding model, the learning task is to increase the similarity between the reference image and the positive image, and decrease the similarity between the reference image and the negative image. That is, to minimize the distance between the graph embedding features of the reference image and the positive image, and to maximize the distance between the graph embedding features of the reference image and the negative image.

[0165] Based on the above learning task, the loss function can be constructed as follows:

[0166] ,

[0167] Where a is the baseline image, p is the positive image, n is the negative image, f is the first image embedding model, and α takes the value of 0.2.

[0168] Based on the aforementioned loss function, the value of the loss function is calculated and backpropagated to each layer of the first image embedding model. Similarly, the loss function value is backpropagated to each layer of the image classification model, and the model parameters of each layer are updated using stochastic gradient descent, thus completing one round of training. Multiple rounds of training can be performed to obtain the second image embedding model.

[0169] In some embodiments, the server may also obtain a test set and input the test set into the first graph embedding model and the second graph embedding model respectively to obtain a first prediction result corresponding to the first graph embedding model and a second prediction result corresponding to the second graph embedding model; based on the first prediction result, determine the value of the loss function corresponding to the first graph embedding model; based on the second prediction result, determine the value of the loss function corresponding to the second graph embedding model; when the value of the loss function corresponding to the second graph embedding model is less than the value of the loss function corresponding to the first graph embedding model, update the first graph embedding model with the second graph embedding model.

[0170] In practice, the first graph embedding model is only updated with the second graph embedding model when the second model is superior, so that subsequent scene segmentation is based on the second graph embedding model. Here, the first and second graph embedding models are tested using a test set, and the value of the loss function is used as a parameter to measure the performance of the models, comparing the first and second graph embedding models.

[0171] In some embodiments, the test set may include a standard test set and a hard test set. Then, the loss function values ​​L00 of the standard test set corresponding to the first graph embedding model, L01 of the hard test set corresponding to the first graph embedding model, L10 of the standard test set corresponding to the second graph embedding model, and L11 of the hard test set corresponding to the second graph embedding model can be obtained. When L00 is greater than L10 and L01 is greater than L11, the first graph embedding model is updated with the second graph embedding model.

[0172] The standard test set is constructed based on uniformly distributed long video series samples, which are obtained through manual cleaning; the difficult test set is constructed based on difficult storyboard examples.

[0173] Here, we take the construction of a standard test set as an example. We obtain standard sample pairs A and B, and randomly select multiple triples, such as selecting 10 triples to form test subset 1. We select once or multiple times to obtain one or more test subsets, thus obtaining the standard test set. For example, selecting 10 times results in a standard test set consisting of 10 test subsets. Here, the construction method for the difficult test set is the same as that for the standard test set.

[0174] It should be noted that the method for extracting triples is the same as the method used when constructing the training set.

[0175] Then, each triplet in the standard test set is input into the first graph embedding model, which outputs the graph embedding features of the baseline image, the positive image, and the negative image in each triplet. Based on these graph embedding features, the value of the loss function is calculated. When the standard test set includes multiple test subsets, the average value of the loss function for the multiple test subsets is taken as L00.

[0176] Accordingly, each triplet in the hard example test set is input into the first graph embedding model, which outputs the graph embedding features of the baseline image, the positive image, and the negative image in each triplet. Based on these graph embedding features, the value of the loss function is calculated. When the hard example test set includes multiple test subsets, the average value of the loss function for multiple test subsets is taken as L01.

[0177] Here, the loss function value L10 of the standard test set of the second graph embedding model and the loss function value L10 of the difficult test set of the second graph embedding model can be obtained in the same way as described above.

[0178] Applying the above embodiments, a video frame sequence corresponding to the target video is obtained by performing frame extraction processing on the target video; features are extracted from each video frame in the video frame sequence using a first graph embedding model to obtain graph embedding features for each video frame; based on the graph embedding features of each video frame, the similarity between the graph embedding features of adjacent video frames in the video frame sequence is obtained; based on the similarity between the graph embedding features of adjacent video frames, the video frame sequence is processed into a storyboard to obtain at least two initial storyboards, each initial storyboard including at least one video frame; compared with SIFT features, the graph embedding features of this application can more accurately describe video frames, thereby improving the accuracy of storyboarding; furthermore, this application aggregates the initial storyboards, resulting in a higher degree of internal aggregation within the target storyboard, thus improving the degree of storyboard aggregation.

[0179] The video scene splitting method provided in the embodiments of this application will be further described below. See also... Figure 9 , Figure 9 This application provides a video scene splitting method, which is implemented collaboratively by a first terminal, a second terminal, and a server. The video scene splitting method includes:

[0180] Step 901: The first terminal sends the target video to the server.

[0181] Step 902: The server performs frame extraction on the target video to obtain the corresponding video frame sequence.

[0182] Step 903: The server extracts features from each video frame in the video frame sequence using the first graph embedding model to obtain the graph embedding features of each video frame.

[0183] Step 904: The server assigns two video frames whose similarity between graph embedding features reaches the first similarity threshold to the same scene, and assigns two video frames whose similarity between graph embedding features does not reach the second similarity threshold to different scenes, so as to obtain at least two initial scenes.

[0184] Step 905: The server obtains the similarity between the initial storyboards.

[0185] Step 906: The server aggregates at least two initial storyboards based on the similarity between the initial storyboards to obtain at least two target storyboards.

[0186] Step 907: Starting from the first video frame in the target storyboard, the server obtains 1 / 3 of the video frames in the order of playback time to obtain the first subset of video frames. Then, starting from the last video frame in the target storyboard, the server obtains 1 / 3 of the video frames in reverse order of playback time to obtain the second subset of video frames.

[0187] Step 908: The server obtains the similarity between the first subset of video frames and the second subset of video frames.

[0188] Step 909: When the similarity does not reach the similarity threshold, the server determines the target storyboard as a difficult storyboard example.

[0189] Step 910: The server sends the storyboard example to the second terminal.

[0190] Step 911: The second terminal obtains the manual storyboard results for the corresponding storyboard difficulty example and returns them to the server.

[0191] Step 912: The server constructs a training set based on the difficult examples of storyboarding and the results of manual storyboarding, and updates the model parameters of the first image embedding model through the training set to obtain the second image embedding model.

[0192] Step 913: The server obtains the test set and inputs the test set into the first graph embedding model and the second graph embedding model respectively to obtain the first prediction result of the first graph embedding model and the second prediction result of the second graph embedding model.

[0193] Step 914: Based on the first prediction result, the server determines the value of the loss function corresponding to the first graph embedding model.

[0194] Step 915: Based on the second prediction result, the server determines the value of the loss function corresponding to the second graph embedding model.

[0195] Step 916: When the value of the loss function corresponding to the second graph embedding model is less than the value of the loss function corresponding to the first graph embedding model, the server updates the first graph embedding model with the second graph embedding model.

[0196] Applying the above embodiments, since the first graph embedding model is obtained through data training, compared with SIFT features, the graph embedding features of this application can more accurately describe video frames, thereby improving the accuracy of the scene segmentation; this application improves the scene segmentation degree by aggregating the initial scene segment, making the internal aggregation degree in the target scene segment higher; and, by obtaining difficult examples of the scene segmentation from the target scene segment to construct a training set, the model parameters of the first graph embedding model are updated, achieving self-optimization and further improving the accuracy of the scene segmentation.

[0197] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario. Figure 10 This is a flowchart illustrating the video scene splitting method provided in this application embodiment. See also... Figure 10 The video scene splitting method provided in this application includes:

[0198] Step 1001: The server retrieves the video.

[0199] Step 1002: The server extracts frames from the video.

[0200] In practice, the open-source image library OpenCV is used to extract frames from the video, converting the video into a video frame sequence.

[0201] For example, for a 10-second video at 25fps, a total of 250 video frames can be extracted.

[0202] Step 1003: The server obtains the graph embedding features of each video frame through a deep learning network model.

[0203] Here, a pre-defined deep learning network model is used to extract features from each video frame to obtain the graph embedding features of each video frame.

[0204] In practical implementation, ResNet101 can be used as the network structure. Here, Tencent's open-source ML-Images (a large-scale image classification dataset) is used to train the model parameters to obtain a preset deep learning network model. Through the preset deep learning network model, features are extracted frame by frame from the above video frame sequence. After each video frame is input into the deep learning network model, the output of the Pool_cr layer in Table 2 is used as the graph embedding feature, i.e., 1*2048-dimensional features.

[0205] Step 1004: The server performs scene segmentation based on the similarity between the graph embedding features of adjacent video frames.

[0206] In actual implementation, a first similarity threshold and a second similarity threshold are preset (e.g., the first similarity threshold is 0.9 and the second similarity threshold is 0.85). A storyboard result dictionary D={1:[],2:[],3:[],4:[],…,M:[]} is established, where 1 represents the first initial storyboard. A list is set under it to store the subsequent storyboard results (a list of corresponding frame IDs). For example, D={1:[1,2,3],2:[4,5],…} means that frames 1, 2, and 3 belong to the first initial storyboard, and frames 4 and 5 belong to the second initial storyboard.

[0207] Store the first video frame in the first initial shot, with the current initial shot set to shotx of value 1 (indicating that the current video frame belongs to the initial shot x). For the second to Nth video frames x, perform the following operations:

[0208] 1) If the similarity between the graph embedding features of video frame x and video frame x-1 is greater than the first similarity (thr1), then x and x-1 belong to the same initial storyboard, and x is recorded in the same storyboard list as x-1.

[0209] 2) If the similarity between the graph embedding features of video frame x and video frame x-1 is less than the second similarity threshold (thr2), then create a new initial storyboard shotx+1 in D and record video frame x in the new storyboard list.

[0210] In the process of segmentation, this application does not retain video frames whose similarity to the graph embedding features of the previous video frame is between thr1 and thr2.

[0211] Step 1005: The server performs aggregation processing on the initial storyboard.

[0212] Here, for each initial segment P from initial segment 1 to initial segment M, the number of video frames n in the initial segment is obtained. For each initial segment q from initial segment P to initial segment M, if the initial segment q has not been merged, it is determined whether the similarity between initial segment p and initial segment q is greater than 0.5. If so, the initial segment p and initial segment q are merged into a target segment; if the initial segment q has already been merged, the initial segment q is skipped.

[0213] The similarity between the initial storyboard p and the initial storyboard q can be obtained in the following way: For each target video frame in the initial storyboard p, obtain the similarity between the target video frame and each video frame in the initial storyboard q, obtain the maximum similarity among the multiple similarities obtained, and take it as the maximum similarity corresponding to the target video frame. Obtain the number of times the maximum similarity of each target video frame reaches the first similarity threshold, nsim, and take nsim / n as the similarity between the initial storyboard p and the initial storyboard q.

[0214] Step 1006: The server extracts difficult examples of storyboards.

[0215] Here, difficult examples of storyboards are obtained from multiple target storyboards based on the length of the target storyboard and the similarity between the inside and outside of the target storyboard.

[0216] In actual implementation, the average number of video frames in the target segment is calculated, which is the total number of video frames / the number of target segments. For each target segment, if the number of video frames in the target segment is less than the average number, or if the number of video frames in the target segment is less than 2, then the target segment is identified as a difficult segment example.

[0217] Alternatively, if the similarity between the first 1 / 3 and the last 1 / 3 of the video frames within a storyboard is less than 0.3, then the first 1 / 3 and the last 1 / 3 are considered difficult examples in each other's storyboards and are recorded as AB storyboard difficult example pairs. In difficult example mining, due to the massive amount of data and limited human resources, this method aims for sufficiently high internal aggregation of the mined difficult examples to avoid secondary labeling caused by improper internal aggregation. Therefore, a smaller threshold (0.3 compared to thr1) is used here to ensure a precise number of difficult examples and accurate results.

[0218] The method for calculating the similarity between video frame subsets is the same as the method for determining the similarity between the initial storyboards.

[0219] Step 1007: The server collects difficult case data.

[0220] Here, the acquired storyboard examples are judged manually, that is, whether the target storyboard is correctly segmented, so as to mark the video frames belonging to the same storyboard.

[0221] For the storyboard pairs obtained above, manually determine whether A and B are different storyboard contents. If so, save the storyboard pair.

[0222] Step 1008: The server accumulates basic video storyboard data.

[0223] Here, video frames belonging to the same storyboard are acquired to accumulate basic storyboard data.

[0224] This application requires maintaining three datasets: a difficult storyboard example test set, a storyboard dataset, and a standard storyboard test set. First, the difficult storyboard example test set is initialized to empty. The storyboard dataset is initialized from uniformly distributed long video series samples P, containing the storyboard results corresponding to the target storyboard. The standard storyboard test set is initialized from uniformly distributed long video series samples Q (which have no intersection with P), containing the standard storyboard test set obtained through manual cleaning of the storyboard results corresponding to the target storyboard. The above difficult storyboard examples are then added to the storyboard database and simultaneously to the difficult storyboard example test set.

[0225] Here, each test set contains A / B hard example pairs: {a1, a2, ...} and {b1, b2, b3, ...}, where a represents the A segment composed of a1, a2, ... and the B segment composed of b1, b2, b3. Images within the same set are similar to each other, while images outside the set are dissimilar to each other. In each training iteration, a triplet (Anchor, Positive, Negative) can be extracted as a sample combination. The Anchor is the baseline image, the Positive is a positive image belonging to the same segment as the baseline image, and the Negative is a negative image belonging to a different segment than the baseline image. For example, (a1, a2, b1) and (a1, a3, b2) can be extracted from A / B. The task of the subsequent deep learning network model is to increase the anchor-positive similarity and decrease the anchor-negative similarity.

[0226] Step 1009: The server trains the deep learning network model.

[0227] Here, model training based on metric learning is achieved through the accumulated storyboard data.

[0228] In practical applications, a gradient descent method based on stochastic gradient descent (SGD) is used to update the convolutional template parameters w and bias parameters b of the deep learning network model. During each iteration, the prediction error is calculated and backpropagated to each layer of the neural network, calculating the gradient and updating the parameters of the convolutional neural network model. Specifically, all model parameters are set to the state to be learned. During training, the deep learning network model performs forward computation on the three input images (Anchor, Positive, Negative) to obtain the prediction result. Based on the prediction result, the loss function is calculated and fed back into the deep learning network model. The model parameters are then updated using stochastic gradient descent, thus achieving parameter optimization. Here, a total of 60 epochs are trained, with each epoch iterating through all A / B sample pairs in the training set (each sample pair uses 10 triples).

[0229] Here, the loss function is Where a is the baseline image, p is the positive image, n is the negative image, f is the deep learning network model, and α takes the value of 0.2.

[0230] Step 1010: The server updates the deep learning network model used to extract graph embedding features.

[0231] Here, the trained model is tested by indicators to see if it meets the update effect. If it does, the deep learning network model is updated so that the video scene can be segmented using the updated deep learning network model.

[0232] In practical applications, firstly, for all standard sample pairs AB in the storyboard standard test set, 10 triples are randomly selected to form test subset 1; this is performed 10 times to form 10 test subsets; for each test subset, a pre-trained neural learning network model is used to extract the graph embedding features of the three images in the triplet, and the value of the loss function is calculated according to the loss function, and the average value of the loss function of all triplets is recorded as Loss.

[0233] Then, the loss function values ​​of the 10 test subsets are taken as the baseline metric L00 corresponding to the standard test set of the storyboard.

[0234] A difficult test set is constructed using similarity metrics, and the corresponding baseline metric L01 is determined. If the difficult test set is empty, then L01 = L00 * 10.

[0235] Next, the updated deep learning network model is used to extract features from the images in the triplets of the standard test set to obtain the corresponding graph embedding features, and the value of the loss function is calculated to obtain the value of the loss function L10 of the deep learning network model for the standard test set; and the updated deep learning network model is used to extract features from the images in the triplets of the hard example test set to obtain the corresponding graph embedding features, and the value of the loss function is calculated to obtain the value of the loss function L11 of the deep learning network model for the standard test set; L00 and L10 are compared, and L01 and L11 are compared. When L00 is greater than L10 and L01 is greater than L11, the pre-trained neural learning network model is replaced with the updated deep learning network model.

[0236] Here, after determining to replace the pre-trained neural learning network model with the updated deep learning network model, the similarity thresholds are iterated from 0.05 to 0.99 (step size 0.01). The recall and precision of 10 triplet subsets of the standard test set are calculated, f1 is calculated, and the similarity threshold that results in the highest f1 is selected to replace the original thr1, and the similarity threshold that results in the highest precision is selected to replace the original thr2. If thr2 is greater than thr1, then thr2 = thr1 - 0.05, to ensure that the relative sizes of the first and second level thresholds can be used to distinguish between the same and different shots.

[0237] After updating the deep learning network model used to extract graph embedding features, the next round of storyboarding and data / model iterations continues.

[0238] In some embodiments, the deep learning network model trained with noise can be deployed on cloud services to provide video storyboard services.

[0239] Figure 11 This is a schematic diagram of the video scene processing flow provided in the embodiments of this application. See also: Figure 11 Terminal A receives the video input by the user and uploads it to the server. The server uses the above method to perform segmentation on the video, extracts difficult examples from the segmentation, outputs the segmentation results to terminal B, and accumulates the segmentation database to update the deep learning network model.

[0240] The embodiments of this application have the following beneficial effects:

[0241] 1) The video segmentation was improved by using a data-driven deep learning network model.

[0242] 2) Weakly supervised active learning method is adopted to extract relatively accurate samples from massive video samples as training set, and a standard test set is established by manual cleaning to measure the model performance.

[0243] 3) Through a closed-loop process of initial storyboarding, difficult case re-entry, model learning and storyboard optimization, the storyboarding system achieves self-optimization, requiring only limited manual cleaning to maintain continuous iteration of the system under massive film and television data.

[0244] The video split-screen device provided in the embodiments of this application will be described below. See also Figure 12 , Figure 12 This is a schematic diagram of the video splitting device provided in an embodiment of this application. The video splitting device provided in an embodiment of this application includes:

[0245] The frame extraction module 210 is used to perform frame extraction processing on the target video to obtain a video frame sequence corresponding to the target video.

[0246] The extraction module 220 is used to extract features from each video frame in the video frame sequence to obtain the graph embedding features of each video frame.

[0247] The acquisition module 230 is used to acquire the similarity between the graph embedding features of adjacent video frames in the video frame sequence based on the graph embedding features of each video frame.

[0248] The storyboard module 240 is used to perform storyboard processing on the video frame sequence based on the similarity between the graph embedding features of the adjacent video frames to obtain at least two initial storyboards, each of the initial storyboards including at least one video frame.

[0249] The aggregation module 250 is used to aggregate at least two initial storyboards to obtain at least two target storyboards.

[0250] In some embodiments, the frame extraction module 210 is further configured to obtain the video frame rate and duration of the target video;

[0251] The number of frames to be extracted is determined based on the target video's frame rate and duration.

[0252] Based on the stated number of frames extracted, the target video is subjected to frame extraction processing to obtain a video frame sequence containing the stated number of video frames extracted.

[0253] In some embodiments, the extraction module is further configured to extract features from each video frame in the video frame sequence using a first graph embedding model to obtain graph embedding features for each video frame.

[0254] The device further includes:

[0255] The update module is used to filter out difficult storyboard examples from at least two target storyboards, wherein the difficult storyboard examples are target storyboards that are judged to be storyboard errors;

[0256] Obtain the manual storyboard result corresponding to the storyboard difficulty example, and the manual storyboard result is used to indicate the video frames belonging to the same storyboard;

[0257] A training set is constructed based on the difficult storyboard examples and the manual storyboard results. The model parameters of the first image embedding model are updated using the training set to obtain the second image embedding model.

[0258] In some embodiments, the updating module is further configured to obtain the number of video frames in each target segment and the average number of video frames corresponding to each target segment;

[0259] Determine the ratio of the number of video frames in each target scene to the average number of video frames;

[0260] A storyboard where the ratio of the number of video frames to the average number of frames does not reach the target ratio is considered a difficult storyboard example.

[0261] In some embodiments, the updating module is further configured to perform the following operations for each target scene when the target scene comprises at least two video frames:

[0262] Starting from the first video frame in the target storyboard, video frames of the target proportion are obtained according to the playback time sequence to obtain the first subset of video frames, and

[0263] Starting from the last video frame in the target segment, video frames of the target proportion are obtained in reverse order according to the playback time points to obtain the second subset of video frames.

[0264] Obtain the similarity between the first subset of video frames and the second subset of video frames;

[0265] When the similarity does not reach the similarity threshold, the target storyboard is determined to be a difficult storyboard example.

[0266] In some embodiments, the training set includes multiple triples, each triple including a baseline image, a positive image belonging to the same shot as the baseline image, and a negative image belonging to a different shot than the baseline image;

[0267] The update module is also used to input the reference image, positive image, and negative image in the triplet into the first image embedding model;

[0268] The first graph embedding model is used to perform forward computation on the reference image, positive image, and negative image to predict the graph embedding features of the reference image, the positive image, and the negative image.

[0269] The first difference between the graph embedding features of the reference image and the graph embedding features of the positive image, and the second difference between the graph embedding features of the reference image and the graph embedding features of the negative image are obtained. Based on the first difference and the second difference, the value of the loss function of the first graph embedding model is determined.

[0270] Based on the value of the loss function, the model parameters of the first graph embedding model are updated.

[0271] In some embodiments, the update module is further configured to obtain a test set and input the test set into a first graph embedding model and a second graph embedding model respectively to obtain a first prediction result corresponding to the first graph embedding model and a second prediction result corresponding to the second graph embedding model;

[0272] Based on the first prediction result, determine the value of the loss function corresponding to the first graph embedding model;

[0273] Based on the second prediction result, determine the value of the loss function corresponding to the second graph embedding model;

[0274] When the value of the loss function corresponding to the second graph embedding model is less than the value of the loss function corresponding to the first graph embedding model, the first graph embedding model is updated using the second graph embedding model.

[0275] In some embodiments, the storyboard module is further configured to obtain a first similarity threshold and a second similarity threshold, wherein the first similarity threshold is greater than the second similarity threshold;

[0276] Two video frames whose similarity between graph embedding features reaches the first similarity threshold are grouped into the same shot.

[0277] And two video frames whose similarity between graph embedding features does not reach the second similarity threshold are divided into different sub-shots to obtain at least two initial sub-shots.

[0278] In some embodiments, the storyboard module 240 is further configured to acquire at least two triplets, the triplets including a reference image, a positive image belonging to the same storyboard as the reference image, and a negative image belonging to a different storyboard than the reference image;

[0279] Obtain at least two candidate similarity thresholds;

[0280] Based on the triplet, the storyboard accuracy and storyboard balance score corresponding to each candidate similarity threshold are determined.

[0281] The candidate similarity threshold with the highest storyboard balance score is used as the first similarity threshold;

[0282] The candidate similarity threshold with the highest storyboard accuracy is used as the second similarity threshold.

[0283] In some embodiments, the aggregation module 250 is further configured to obtain the similarity between initial storyboards;

[0284] Based on the similarity between the initial storyboards, the at least two initial storyboards are aggregated so that the similarity between any two initial storyboards aggregated into the same target storyboard reaches a similarity threshold.

[0285] In some embodiments, the aggregation module 250 is further configured to, when the initial scene includes at least two video frames, use the first initial scene and the second initial scene as any two of the at least two initial scenes, and perform the following processing on any two initial scenes:

[0286] For any target video frame in the first initial storyboard, the similarity between the target video frame and each video frame in the second initial storyboard is obtained, and at least two similarities are obtained for the target video frame.

[0287] Based on at least two similarities corresponding to each target video frame in the first initial segment, determine the maximum similarity corresponding to each target video frame in the first initial segment;

[0288] Obtain the first number of target video frames whose maximum similarity reaches the first similarity threshold, and the second number of target video frames in the first initial storyboard;

[0289] The ratio of the first quantity to the second quantity is used as the similarity between the first initial storyboard and the second initial storyboard.

[0290] In some embodiments, the aggregation module 240 is further configured to, when the initial scene includes at least two video frames, use the first initial scene and the second initial scene as any two of the at least two initial scenes, and perform the following processing on any two initial scenes:

[0291] When the playback time sequence of the first initial scene is earlier than that of the second initial scene, the target video frame with the last playback time sequence in the first initial scene is obtained.

[0292] Obtain the similarity between the target video frame and each video frame in the second initial storyboard, and obtain at least two similarities corresponding to the target video frame;

[0293] The average similarity of at least two similarities is used as the similarity between the first initial storyboard and the second initial storyboard.

[0294] Applying the above embodiments, compared with SIFT features, the graph embedding features of this application can more accurately describe video frames, thereby improving the accuracy of scene segmentation; furthermore, by aggregating the initial scene segmentation, this application makes the internal aggregation degree in the target scene segmentation higher, thereby improving the scene segmentation aggregation degree.

[0295] This application also provides a computer device, which may be a terminal or a server, see [link]. Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device provided in an embodiment of this application includes:

[0296] Memory 550 is used to store executable instructions;

[0297] The processor 510 is configured to execute executable instructions stored in the memory to implement the video playback method provided in the embodiments of this application.

[0298] Here, processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0299] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 510.

[0300] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0301] In some embodiments, at least one network interface 520 and a user interface 530 may also be included. Various components in the computer device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 13 The general labeled all buses as Bus System 540.

[0302] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video scene splitting method described in this application.

[0303] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 4 The method shown.

[0304] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0305] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0306] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0307] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0308] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A video scene segmentation method, characterized in that, The method includes: Frame extraction is performed on the target video to obtain a video frame sequence corresponding to the target video; The first graph embedding model is used to extract features from each video frame in the video frame sequence to obtain the graph embedding features of each video frame. Based on the graph embedding features of each video frame, the similarity between the graph embedding features of adjacent video frames in the video frame sequence is obtained. Based on the similarity between the graph embedding features of the adjacent video frames, the video frame sequence is processed into a split-scene format to obtain at least two initial split-scenes, each of which includes at least one video frame. The at least two initial storyboards are aggregated to obtain at least two target storyboards; Select difficult storyboard examples from the at least two target storyboard examples; obtain the manual storyboard results corresponding to the difficult storyboard examples, the manual storyboard results being used to indicate video frames belonging to the same storyboard example; A training set is constructed based on the difficult storyboard examples and the manual storyboard results. The model parameters of the first image embedding model are updated using the training set to obtain the second image embedding model.

2. The method as described in claim 1, characterized in that, The step of extracting frames from the target video to obtain a video frame sequence corresponding to the target video includes: Obtain the video frame rate and duration of the target video; The number of frames to be extracted is determined based on the video frame rate and the duration of the target video. Based on the stated number of frames extracted, the target video is subjected to frame extraction processing to obtain a video frame sequence containing the stated number of video frames extracted.

3. The method as described in claim 1, characterized in that, The process of selecting difficult storyboard examples from the at least two target storyboards includes: Obtain the number of video frames in each target scene and the average number of video frames corresponding to each target scene; Determine the ratio of the number of video frames in each target scene to the average number of video frames; A storyboard where the ratio of the number of video frames to the average number of video frames does not reach the target ratio is considered a difficult storyboard example.

4. The method as described in claim 1, characterized in that, The process of selecting difficult storyboard examples from the at least two target storyboards includes: When the target scene comprises at least two video frames, perform the following operations for each target scene: Starting from the first video frame in the target storyboard, video frames of the target proportion are obtained according to the playback time sequence to obtain the first subset of video frames, and Starting from the last video frame in the target segment, video frames of the target proportion are obtained in reverse order according to the playback time points to obtain the second subset of video frames. Obtain the similarity between the first subset of video frames and the second subset of video frames; When the similarity does not reach the similarity threshold, the target storyboard is determined to be a difficult storyboard example.

5. The method as described in claim 1, characterized in that, The training set includes multiple triples, each triple including a baseline image, a positive image belonging to the same shot as the baseline image, and a negative image belonging to a different shot than the baseline image; The step of updating the model parameters of the first graph embedding model using the training set includes: The reference image, positive image, and negative image in the triplet are input into the first image embedding model; The first graph embedding model is used to perform forward computation on the reference image, positive image, and negative image to predict the graph embedding features of the reference image, the positive image, and the negative image. Obtain the first difference between the graph embedding features of the reference image and the graph embedding features of the positive image, and the second difference between the graph embedding features of the reference image and the graph embedding features of the negative image. Based on the first difference and the second difference, the value of the loss function of the first graph embedding model is determined; Based on the value of the loss function, the model parameters of the first graph embedding model are updated.

6. The method as described in claim 1, characterized in that, After updating the model parameters of the first graph embedding model, the method further includes: A test set is obtained and input into the first graph embedding model and the second graph embedding model respectively to obtain a first prediction result corresponding to the first graph embedding model and a second prediction result corresponding to the second graph embedding model. Based on the first prediction result, determine the value of the loss function corresponding to the first graph embedding model; Based on the second prediction result, determine the value of the loss function corresponding to the second graph embedding model; When the value of the loss function corresponding to the second graph embedding model is less than the value of the loss function corresponding to the first graph embedding model, the first graph embedding model is updated using the second graph embedding model.

7. The method as described in claim 1, characterized in that, The video frame sequence is processed by segmentation based on the similarity between the graph embedding features of adjacent video frames to obtain at least two initial segmentations, including: Obtain a first similarity threshold and a second similarity threshold, wherein the first similarity threshold is greater than the second similarity threshold; Two video frames whose similarity between graph embedding features reaches the first similarity threshold are grouped into the same shot. And two video frames whose similarity between graph embedding features does not reach the second similarity threshold are divided into different sub-shots to obtain at least two initial sub-shots.

8. The method as described in claim 7, characterized in that, The process of obtaining the first similarity threshold and the second similarity threshold includes: Obtain at least two triplets, wherein the triplets include a reference image, a positive image belonging to the same shot as the reference image, and a negative image belonging to a different shot than the reference image; Obtain at least two candidate similarity thresholds; Based on the triplet, the storyboard accuracy and storyboard balance score corresponding to each candidate similarity threshold are determined. The candidate similarity threshold with the highest storyboard balance score is used as the first similarity threshold; The candidate similarity threshold with the highest storyboard accuracy is used as the second similarity threshold.

9. The method as described in claim 1, characterized in that, The aggregation process of the at least two initial storyboards to obtain at least two target storyboards includes: Obtain the similarity between the initial storyboards; Based on the similarity between the initial storyboards, the at least two initial storyboards are aggregated so that the similarity between any two initial storyboards aggregated into the same target storyboard reaches a similarity threshold.

10. The method as described in claim 9, characterized in that, The process of obtaining the similarity between the initial storyboards includes: When the initial storyboard includes at least two video frames, the first initial storyboard and the second initial storyboard are used as any two initial storyboards among the at least two initial storyboards, and the following processing is performed on any two initial storyboards: For any target video frame in the first initial storyboard, the similarity between the target video frame and each video frame in the second initial storyboard is obtained, and at least two similarities are obtained for the target video frame. Based on at least two similarities corresponding to each target video frame in the first initial segment, determine the maximum similarity corresponding to each target video frame in the first initial segment; Obtain the first number of target video frames whose maximum similarity reaches the third similarity threshold, and the second number of target video frames in the first initial storyboard; The ratio of the first quantity to the second quantity is used as the similarity between the first initial storyboard and the second initial storyboard.

11. The method as described in claim 9, characterized in that, The process of obtaining the similarity between the initial storyboards includes: When the initial storyboard includes at least two video frames, the first initial storyboard and the second initial storyboard are used as any two initial storyboards among the at least two initial storyboards, and the following processing is performed on any two initial storyboards: When the playback time sequence of the first initial scene is earlier than that of the second initial scene, the target video frame with the last playback time sequence in the first initial scene is obtained. Obtain the similarity between the target video frame and each video frame in the second initial storyboard, and obtain at least two similarities corresponding to the target video frame; The average similarity of at least two similarities is used as the similarity between the first initial storyboard and the second initial storyboard.

12. A video split-screen device, characterized in that, include: The frame extraction module is used to extract frames from the target video to obtain a video frame sequence corresponding to the target video. The extraction module is used to extract features from each video frame in the video frame sequence using a first graph embedding model to obtain the graph embedding features of each video frame. The acquisition module is used to acquire the similarity between the graph embedding features of adjacent video frames in the video frame sequence based on the graph embedding features of each video frame. The storyboard module is used to perform storyboard processing on the video frame sequence based on the similarity between the graph embedding features of the adjacent video frames to obtain at least two initial storyboards, each of the initial storyboards including at least one video frame. The aggregation module is used to aggregate at least two initial storyboards to obtain at least two target storyboards; The update module is used to filter out difficult storyboard examples from the at least two target storyboards; obtain the manual storyboard results corresponding to the difficult storyboard examples, wherein the manual storyboard results are used to indicate video frames belonging to the same storyboard; A training set is constructed based on the difficult storyboard examples and the manual storyboard results. The model parameters of the first image embedding model are updated using the training set to obtain the second image embedding model.

13. The apparatus according to claim 12, characterized in that, The frame extraction module is also used to obtain the video frame rate of the target video and the duration of the target video; The number of frames to be extracted is determined based on the video frame rate and the duration of the target video. Based on the stated number of frames extracted, the target video is subjected to frame extraction processing to obtain a video frame sequence containing the stated number of video frames extracted.

14. The apparatus according to claim 12, characterized in that, The update module is also used to obtain the number of video frames in each target segment and the average number of video frames corresponding to each target segment; Determine the ratio of the number of video frames in each target scene to the average number of video frames; A storyboard where the ratio of the number of video frames to the average number of video frames does not reach the target ratio is considered a difficult storyboard example.

15. The apparatus according to claim 12, characterized in that, The update module is further configured to perform the following operations for each target scene when the target scene comprises at least two video frames: Starting from the first video frame in the target storyboard, video frames of the target proportion are obtained according to the playback time sequence to obtain the first subset of video frames, and Starting from the last video frame in the target segment, video frames of the target proportion are obtained in reverse order according to the playback time points to obtain the second subset of video frames. Obtain the similarity between the first subset of video frames and the second subset of video frames; When the similarity does not reach the similarity threshold, the target storyboard is determined to be a difficult storyboard example.

16. The apparatus according to claim 12, characterized in that, The training set includes multiple triples, each triple including a baseline image, a positive image belonging to the same shot as the baseline image, and a negative image belonging to a different shot than the baseline image; The update module is also used to input the reference image, positive image, and negative image in the triplet into the first image embedding model; The first graph embedding model is used to perform forward computation on the reference image, positive image, and negative image to predict the graph embedding features of the reference image, the positive image, and the negative image. Obtain the first difference between the graph embedding features of the reference image and the graph embedding features of the positive image, and the second difference between the graph embedding features of the reference image and the graph embedding features of the negative image. Based on the first difference and the second difference, the value of the loss function of the first graph embedding model is determined; Based on the value of the loss function, the model parameters of the first graph embedding model are updated.

17. The apparatus according to claim 12, characterized in that, The update module is further configured to obtain a test set and input the test set into the first graph embedding model and the second graph embedding model respectively to obtain a first prediction result corresponding to the first graph embedding model and a second prediction result corresponding to the second graph embedding model; Based on the first prediction result, determine the value of the loss function corresponding to the first graph embedding model; Based on the second prediction result, determine the value of the loss function corresponding to the second graph embedding model; When the value of the loss function corresponding to the second graph embedding model is less than the value of the loss function corresponding to the first graph embedding model, the first graph embedding model is updated using the second graph embedding model.

18. A computer device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the video splitting method according to any one of claims 1 to 11.

19. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the video splitting method according to any one of claims 1 to 11 when executed by a processor.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the video splitting method according to any one of claims 1 to 11 is implemented.

Images