Video scene segmentation method, apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose a video scene segmentation method, device, equipment and storage medium. The method comprises: performing video segmentation on the video to obtain a plurality of shot pictures; performing feature extraction on the plurality of shot pictures respectively to obtain multi-modal features of each shot picture in the plurality of shot pictures; performing fusion processing on the multi-modal features of each shot picture in the plurality of shot pictures respectively to obtain fusion semantic information of each shot picture in the plurality of shot pictures; determining scene segmentation positions in the plurality of shot pictures according to the fusion semantic information of each shot picture in the plurality of shot pictures, and performing scene segmentation on the plurality of shot pictures. By using the method, the accuracy of video scene segmentation is improved.

Description

Technical Field

[0001] This invention relates to the field of video segmentation, and more particularly to a method, apparatus, device, and storage medium for video scene segmentation. Background Technology

[0002] In recent years, with the production of massive amounts of video, video understanding tasks have become particularly important. Video understanding encompasses many fundamental capabilities, including video segmentation, video classification, and video information extraction. Therefore, the effectiveness of video understanding tasks directly determines downstream tasks, including automatic video tagging, video search, and video recommendation, playing a crucial role. Video scene segmentation is a very important subtask within video understanding. The goal of video scene segmentation is to divide long videos into continuous parts with invariant visuals or audio, with multiple consecutive shots forming an information unit. For example, in a news broadcast video, a long video can be segmented into three scenes: the first shot is the live broadcast scene, the second and third shots are outdoor segments, and the fourth shot is the live broadcast scene again. The segmentation between the first and second shots represents the dividing point between two scenes; similarly, the segmentation between the third and fourth shots results in three scenes.

[0003] The core issue in scene segmentation is finding the segmentation points between scenes. Mainstream methods are divided into unsupervised and supervised methods. Unsupervised methods primarily rely on calculating the similarity matrix between shots to find segmentation points. Supervised methods are mainly based on deep learning, particularly Siamese networks. The problem with unsupervised methods is their heavy dependence on parameter design, resulting in low accuracy and poor generalization ability; supervised methods heavily rely on labeled data, which is costly and has limited scalability.

[0004] Currently, unsupervised algorithms lack supervisory information, have poor discriminative ability, and are not stable enough in performance; supervised methods rely heavily on supervised samples, however, the cost of video annotation is too high and samples are very scarce; therefore, a new video scene segmentation method is needed. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for video scene segmentation, which can improve the accuracy of scene segmentation.

[0006] In a first aspect, embodiments of the present invention provide a method for video scene segmentation, comprising:

[0007] The video is segmented to obtain multiple shot frames;

[0008] Feature extraction is performed on each of the multiple shot frames to obtain the multimodal features of each shot frame in the multiple shot frames;

[0009] The multimodal features of each shot in the multiple shot frames are fused separately to obtain the fused semantic information of each shot in the multiple shot frames;

[0010] The scene segmentation position in the multiple shot frames is determined based on the fused semantic information of each shot frame in the multiple shot frames, and the multiple shot frames are then segmented into scenes.

[0011] As an optional implementation, determining the scene segmentation position in the multiple shot frames based on the fused semantic information of each shot frame, and performing scene segmentation on the multiple shot frames, includes:

[0012] The difference value between each shot and its adjacent shot is determined based on the fusion semantic information of each shot in the plurality of shot frames;

[0013] Based on the difference between each shot and its adjacent shot, the scene segmentation position in the plurality of shot images is determined, and the plurality of shot images are segmented.

[0014] As an optional implementation, determining the difference value between each shot and its adjacent shot based on the fusion semantic information of each shot in the plurality of shot frames includes:

[0015] The first set of shots and the second set of shots are determined based on the current shot. The first set of shots consists of the current shot and the K consecutive shots preceding it, and the second set of shots consists of the K consecutive shots following the current shot.

[0016] The fusion semantic information of the shots contained in the first group of shots and the fusion semantic information of the shots contained in the second group of shots are respectively convolved to obtain a first fusion semantic vector and a second fusion semantic vector. The first fusion semantic vector corresponds to the fusion semantic information of the shots contained in the first group of shots, and the second fusion semantic vector corresponds to the fusion semantic information of the shots contained in the second group of shots.

[0017] The difference value between the current shot and its adjacent shot is determined based on the first fusion semantic vector and the second fusion semantic vector, wherein the difference value between each of the plurality of shot images and its adjacent shot images includes the difference value between the current shot and its adjacent shot images.

[0018] As an optional implementation, determining the difference value between the current shot and its adjacent shot based on the first fusion semantic vector and the second fusion semantic vector includes:

[0019] The difference distance is calculated based on the first fused semantic vector and the second fused semantic vector;

[0020] The difference value between the current shot and its adjacent shot is determined based on the difference distance.

[0021] The step of determining the scene segmentation position in the plurality of shot frames based on the difference value between each shot frame and its adjacent shot frames includes:

[0022] If the difference between the current shot and its adjacent shot is greater than a preset value, then the position between the current shot and its adjacent shot is determined as the segmentation position.

[0023] If the difference between the current shot and its adjacent shot is not greater than the preset value, then the current shot and its adjacent shot are determined to be shots of the same scene.

[0024] As an optional implementation, the multimodal features of each shot include at least two of the following: speech features, text features, and image features.

[0025] Secondly, embodiments of the present invention provide a video scene segmentation apparatus, comprising:

[0026] The video segmentation module is used to segment the video to obtain multiple shot frames;

[0027] The extraction module is used to extract features from the multiple shot images to obtain the multimodal features of each shot image in the multiple shot images;

[0028] The fusion module is used to perform fusion processing on the multimodal features of each shot in the multiple shot images to obtain the fused semantic information of each shot in the multiple shot images;

[0029] The segmentation module is used to determine the scene segmentation position in the multiple shot frames based on the fusion semantic information of each shot frame in the multiple shot frames, and to perform scene segmentation on the multiple shot frames.

[0030] Thirdly, embodiments of the present invention provide a computer program product, including a computer-readable storage medium storing a computer program, which, when executed by at least one processor, causes the at least one processor to perform the video scene segmentation method as described in the first aspect.

[0031] Fourthly, embodiments of the present invention provide a computer storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, perform the video scene segmentation method as described in the first aspect.

[0032] This application's embodiments segment the video into multiple shot frames, then perform multimodal feature extraction, and fuse the multimodal features of each shot frame separately. Based on the fused semantic information of the shot frames, the scene segmentation location is determined, thus achieving video scene segmentation. This scheme combines the advantages of unsupervised algorithms, which extract discriminative features of shots without relying on labeled samples, with the advantages of supervised algorithms, utilizing pre-trained models of deep learning algorithms to extract multimodal features. These features are then fused online to fully represent the shots, and the scene segmentation location is determined based on the fused semantic information of the shot frames. This approach improves the accuracy of video scene segmentation. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1a This is a schematic diagram illustrating an application scenario of a video scene segmentation method provided in an embodiment of the present invention;

[0035] Figure 1b This is a schematic diagram of a video scene segmentation result provided in an embodiment of the present invention;

[0036] Figure 2 This is a flowchart illustrating a video scene segmentation method provided in an embodiment of the present invention;

[0037] Figure 3 This is a flowchart illustrating another video scene segmentation method provided in an embodiment of the present invention;

[0038] Figure 4a This is a flowchart illustrating another video scene segmentation method provided in an embodiment of the present invention;

[0039] Figure 4b This is a schematic diagram illustrating the calculation of differences in video camera footage provided by an embodiment of the present invention;

[0040] Figure 4c This is a schematic diagram of a video scene segmentation method provided in an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of the structure of a video scene segmentation device provided in an embodiment of the present invention;

[0042] Figure 6 This is a schematic diagram of the structure of another video scene segmentation device provided in an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0044] It should be understood that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0045] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention can be combined with other embodiments.

[0046] Video scene segmentation is a crucial subtask within video understanding. Its goal is to divide long videos into continuous segments where the visuals or audio remain unchanged. These continuous segments are information units composed of multiple consecutive shots (i.e., belonging to a scene). A single shot represents a continuous sequence of images captured by the same camera in a single shot. Since a scene consists of multiple consecutive shots, a shot is an atomic unit in video processing. This solution achieves scene segmentation by segmenting the video into multiple shots, extracting features from these shots, and finally, using these features as the basis for the segmentation.

[0047] Please see Figure 1a The diagram shown is an application scenario illustration of a video scene segmentation method provided by an embodiment of the present invention. Figure 1aAs shown, video server 10d receives video from an electronic device, thereby achieving video scene segmentation. This electronic device may include: computer 10a, laptop 10b, ..., mobile phone 10c, television, etc. Of course, video server 10d can also obtain the video to be segmented through other means. For example, video server 10d receives video sent by laptop 10b, and this video is as follows... Figure 1b The video footage of the ball game shown. Figure 1b As shown, the video includes footage of a badminton match, guest commentary on the badminton match, and basketball match footage. The video server 10d in this solution first divides the video into three segments: footage of the badminton match, footage of the guest commentary on the badminton match, and footage of the basketball match. Then, multimodal feature extraction is performed on each of these three segments, including text features, image features, and speech features. Next, the video server 10d fuses the multimodal features of each segment to obtain fused semantic information for each segment. This fused semantic information is then used to determine the scene segmentation location, such as... Figure 1b The video is divided into three scenes based on the positions indicated by the thick lines. This embodiment only illustrates the example of one scene containing one shot; it should be noted that this solution is not limited to this.

[0048] The video scene segmentation method provided in this application can be applied to multiple video tasks. Besides scene segmentation, it can also be applied to behavior recognition, segment splitting, and other fields. Using this solution can improve the algorithm's performance, thereby enhancing the product experience.

[0049] This solution can be applied to the video production and cataloging stages, providing materials for both processes. For example, if an interview video of a leader is needed, the scene segmentation capability can be used to obtain the corresponding scene clips.

[0050] This solution can also be applied to video archiving and management, automatically archiving different categories of video clips. Through behavior recognition capabilities, it can automatically archive video clips.

[0051] This approach fully integrates the advantages of supervised and unsupervised methods, without relying on labeled data. It utilizes a pre-trained deep network trained with supervised methods to extract multimodal features. By extracting the differences between shots, it identifies scene segmentation locations. This method can also be extended to behavior segmentation problems. This approach combines image, speech, and text multimodal features, fusing them online to fully represent shots. Simultaneously, based on the multimodal fusion features, it explicitly extracts the differences between shots, using the most significant differences as scene segmentation locations. This method improves the accuracy of scene segmentation.

[0052] Please see Figure 2 , Figure 2 This is a flowchart illustrating a video scene segmentation method provided in an embodiment of the present invention; the method includes steps 201-204, as detailed below:

[0053] 201. Perform video segmentation on the video to obtain multiple shot frames;

[0054] A video typically contains multiple scenes, and a scene can contain multiple consecutive shots.

[0055] Long videos are segmented into individual shot units. One implementation method is to use the Scene Change Detection command built into the software ffmpeg. ffmpeg is an open-source computer program that can record, convert, and stream digital audio and video. It provides a complete solution for recording, converting, and streaming audio and video. Specifically, using the following command: `ffmpeg -loglevel error -yi input_video -vf select = gte(scene\,0),metadata = print:file = temp_file_name.txt -an -f null` to segment the video will automatically call ffmpeg's built-in Scene Change Detection algorithm. The main principle of this algorithm is to find the changes in the image between two adjacent frames of the video; if the change is significant at a certain point, the video at that point will be segmented.

[0056] 202. Perform feature extraction on the multiple shot frames respectively to obtain the multimodal features of each shot frame in the multiple shot frames;

[0057] Based on the multiple shot images obtained from the above segmentation, multimodal features are extracted for each shot image.

[0058] The aforementioned multimodal features include at least two of the following: speech features, text features, and image features. Of course, other features are also possible, and this solution does not impose any specific limitations on them.

[0059] For each shot, text, speech, and image features are extracted separately. The modal information that plays a decision-making role differs in different scenarios and at different stages. For example, in a scene where a person is being interviewed, any shot of the interviewer is considered part of the same scene, so sound plays a decisive role here. Conversely, in a studio scene, any image appearing in the studio belongs to this scene, so imagery plays a decisive role here.

[0060] For example, speech features, text features, and image features can be extracted using deep learning pre-trained models. Specifically, speech features can be extracted using VGGish, image features can be extracted using ResNet-VLAD, and text features can be extracted using BERT, etc.

[0061] Deep learning algorithms are trained in a supervised manner on massive datasets, thus enabling them to extract semantic information from various modalities more effectively.

[0062] The above is just one example; other methods can also be used for feature extraction, and this solution does not impose any specific limitations on them.

[0063] 203. Perform fusion processing on the multimodal features of each shot in the multiple shot frames to obtain the fused semantic information of each shot in the multiple shot frames;

[0064] The multimodal features of each extracted shot are then fused, for example, by simple overlay or by other fusion methods, to obtain the fused semantic information of each shot.

[0065] 204. Determine the scene segmentation position in the multiple shot images based on the fusion semantic information of each shot image in the multiple shot images, and perform scene segmentation on the multiple shot images.

[0066] Then, the scene segmentation positions in the multiple shot frames are determined based on the fused semantic information of each shot frame. For example, the scene similarity between two adjacent shots can be calculated to determine whether the two shot frames belong to the same scene. If they belong to the same scene, the two shot frames are not segmented; if they do not belong to the same scene, the two shot frames are identified as a scene segmentation position. By performing the above judgment on each shot frame, multiple scene segmentation positions in the video are determined, thereby achieving scene segmentation of the video.

[0067] This application's embodiments segment the video into multiple shot frames, then perform multimodal feature extraction, and fuse the multimodal features of each shot frame separately. Based on the fused semantic information of the shot frames, the scene segmentation location is determined, thus achieving video scene segmentation. This scheme combines the advantages of unsupervised algorithms, which extract discriminative features of shots without relying on labeled samples, with the advantages of supervised algorithms, utilizing pre-trained models of deep learning algorithms to extract multimodal features. These features are then fused online to fully represent the shots, and the scene segmentation location is determined based on the fused semantic information of the shot frames. This approach improves the accuracy of video scene segmentation.

[0068] Please see Figure 3 , Figure 3 This is a flowchart illustrating a video scene segmentation method provided in an embodiment of the present invention; the method includes steps 301-305, as detailed below:

[0069] 301. Perform video segmentation on the video to obtain multiple shot frames;

[0070] A video typically contains multiple scenes, and a scene can contain multiple consecutive shots.

[0071] This is achieved by segmenting long videos into individual shot units. One implementation method is to use the Scene Change Detection command built into the ffmpeg software to perform the segmentation.

[0072] 302. Perform feature extraction on the multiple shot frames respectively to obtain the multimodal features of each shot frame in the multiple shot frames;

[0073] The aforementioned multimodal features include at least two of the following: speech features, text features, and image features. Of course, other features are also possible, and this solution does not impose any specific limitations on them.

[0074] As one approach, speech features, text features, and image features are extracted using a deep learning pre-trained model.

[0075] For example, the VGGish model is used to extract speech features from multiple camera shots. This VGGish model is pre-trained on YouTube's AudioSet data.

[0076] For example, the ResNet-VLAD residual network model can be used to extract image features from multiple shots. By inputting each shot into the ResNet-VLAD model, image features can be extracted separately.

[0077] Furthermore, the BERT language representation model is used to extract text features from multiple shots. By inputting each shot into the BERT model, text features can be extracted separately.

[0078] As an example, for a video containing news broadcasts and weather reports, the video can be segmented using the Scene Change Detection command built into the software ffmpeg, resulting in multiple shot frames, such as shots of the announcer broadcasting in the studio, shots of interviewers outside the studio, and shots of the weather forecaster giving the weather forecast.

[0079] Then, by inputting the above multiple camera shots into the VGGish model, speech features can be extracted for each camera shot. For example, for the camera shot of the announcer broadcasting in the live broadcast room, the audio "Today's news broadcast is over. Welcome to watch tomorrow" can be extracted, and for the camera shot of the weather forecaster giving a weather forecast, the audio "There will be light to moderate rain in some areas" can be extracted, etc.

[0080] By inputting the aforementioned multiple camera shots into the Resnet-VLAD model, image features can be extracted for each camera shot. For example, for a camera shot of an announcer broadcasting in the studio, the live broadcast image of the announcer broadcasting can be extracted; for a camera shot of an interviewer outside the studio, the image of the interviewer conducting the interview can be extracted, and so on.

[0081] Meanwhile, by inputting the aforementioned multiple camera shots into the BERT model, text features can be extracted for each shot. For example, subtitle information or other text information in the video can be extracted. For example, for a shot of a news anchor broadcasting in the studio, the text information extracted is: "Today's news broadcast is over. Welcome to watch tomorrow." For a shot of a weather forecaster giving a weather forecast, the text information extracted is: "Light to moderate rain in some areas," and so on.

[0082] The above is just one example; other methods can also be used for feature extraction, and this solution does not impose any specific limitations on them.

[0083] 303. Perform fusion processing on the multimodal features of each shot in the multiple shot frames to obtain the fused semantic information of each shot in the multiple shot frames;

[0084] The multimodal features of each extracted shot are then fused, for example, by simple superposition or by other methods, such as superposition based on preset weights, to obtain the fused semantic information of each shot.

[0085] Specifically, the speech features, text features, and image features of any shot A are vector-superimposed. Alternatively, based on preset weights, each feature is multiplied by its corresponding weight before being superimposed as a vector. Of course, other methods are also possible, and this solution does not impose specific limitations on them.

[0086] For example, the speech, text, and image features of any shot A can be concatenated using `concat`. Alternatively, the speech, text, and image features of any shot A can be used to perform vector operations using algorithms such as addition, subtraction, multiplication, and division. Another option is to use attention-based methods for fusion, such as processing based on the Transformer model or the SeNet model.

[0087] 304. Determine the difference value between each shot and its adjacent shot based on the fusion semantic information of each shot in the plurality of shot frames;

[0088] For example, the similarity of the scene between two adjacent shots can be calculated to determine the difference between the two shots.

[0089] The above is just one example. The difference between two shots can also be calculated using other methods. This solution does not impose any specific limitations on this.

[0090] 305. Based on the difference value between each shot and its adjacent shot, determine the scene segmentation position in the plurality of shot images, and perform scene segmentation on the plurality of shot images.

[0091] The difference between a shot and its adjacent shots determines whether they belong to the same scene. If they belong to the same scene, the two shots are not separated; if they do not belong to the same scene, the point between them is designated as a scene segmentation point. By performing this determination on each shot, multiple scene segmentation points in the video are identified, thus achieving scene segmentation of the video.

[0092] This application's embodiments segment a video into multiple frames, then perform multimodal feature extraction, and fuse the multimodal features of each frame separately. Based on the fused semantic information of the frames, the difference between them is calculated to determine the scene segmentation location, thus achieving scene segmentation of the video. This scheme combines the advantages of unsupervised algorithms, which extract discriminative features of frames without relying on labeled samples, with the advantages of supervised algorithms, utilizing a pre-trained model of deep learning to extract multimodal features and determine the difference between frames based on the fused semantic information, thereby determining the scene segmentation location. This approach improves the accuracy of video scene segmentation.

[0093] Please see Figure 4a , Figure 4a This is a flowchart illustrating a video scene segmentation method provided in an embodiment of the present invention; the method includes steps 401-409, as detailed below:

[0094] 401. Perform video segmentation on the video to obtain multiple shot frames;

[0095] For details, please refer to the foregoing embodiments; they will not be repeated here.

[0096] 402. Perform feature extraction on each of the multiple shot frames to obtain the multimodal features of each shot frame in the multiple shot frames;

[0097] For details on this introduction, please refer to the foregoing embodiments; they will not be repeated here.

[0098] 403. Perform fusion processing on the multimodal features of each shot in the multiple shot frames to obtain the fused semantic information of each shot in the multiple shot frames;

[0099] Speech, text, and image features are fused using simple concatenation to obtain the final fused features of the shot. For details on this, please refer to the foregoing embodiments; further elaboration will not be repeated here.

[0100] 404. Determine the first group of frames and the second group of frames based on the current shot, wherein the current shot and the K consecutive shots preceding it are determined as the first group of frames, and the K consecutive shots following the current shot are determined as the second group of frames.

[0101] For example, for shot j, the k consecutive shots preceding shot j form the first set of frames, and the k consecutive shots following shot j form the second set of frames. j can be any integer, such as 1, 2, 5, etc., and this scheme does not impose any specific restrictions on it.

[0102] In particular, for continuous shot sequences such as the first or last shot, empty shot sequences can be used to fill the gaps.

[0103] Specifically, when k is 5, for the first shot, since there are no consecutive shots preceding it, the k consecutive shots preceding it are set to consist of 4 empty shots and the first shot.

[0104] Accordingly, for the last shot, since there are no subsequent consecutive shots, the k consecutive shots following it are set to consist of 5 empty shots.

[0105] 405. Perform convolution processing on the fusion semantic information of the shot images contained in the first group of images and the fusion semantic information of the shot images contained in the second group of images respectively to obtain a first fusion semantic vector and a second fusion semantic vector. The first fusion semantic vector corresponds to the fusion semantic information of the shot images contained in the first group of images, and the second fusion semantic vector corresponds to the fusion semantic information of the shot images contained in the second group of images.

[0106] The fusion semantic information corresponding to k consecutive shots in the first set of images is convolved to obtain the first fusion semantic vector; the fusion semantic information corresponding to k consecutive shots in the second set of images is convolved to obtain the second fusion semantic vector.

[0107] For example, statistical or vector processing methods can be used to obtain the statistics of k shots as the first fusion semantic vector and the second fusion semantic vector.

[0108] Specifically, for the video frames corresponding to the k consecutive shots in the first group of images, the feature dimension extracted from each frame is M, so they can be combined into a matrix of dimension k*M. The matrix can be fused with algorithms such as MaxPooling, AveragePooling or convolution to achieve intra-group information fusion.

[0109] Specifically, the MaxPooling algorithm calculates the maximum value among the k features in the above k*M matrix, and obtains the fused intra-group feature dimension of 1*M based on this maximum value;

[0110] AveragePooling algorithm: For the above k*M matrix, calculate the average of k features, and the dimension of the within-group features after fusion becomes 1*M;

[0111] Convolution operation: conv(input=K*M,kernel_size=(K,1)), the dimension of the intra-group features after fusion becomes 1*M.

[0112] 406. Calculate the difference distance based on the first fused semantic vector and the second fused semantic vector;

[0113] like Figure 4b As shown, the difference distance can be obtained, for example, by calculating the cosine distance between the first fused semantic vector and the second fused semantic vector. Specifically, the cosine distance can be expressed as:

[0114] cosin = X1·X2 / |X1||X2|;

[0115] Wherein, X1 and X2 are the first fused semantic vector and the second fused semantic vector, respectively.

[0116] Of course, the difference distance can also be calculated using other methods, but this solution does not impose any specific limitations on this.

[0117] 407. Determine the difference value between the current shot and its adjacent shot based on the difference distance;

[0118] The difference distance is used to represent the difference between the current shot and its adjacent shots.

[0119] 408. If the difference between the current shot and its adjacent shot is greater than a preset value, then the position between the current shot and its adjacent shot is determined as the segmentation position.

[0120] When the difference value is greater than the preset value, it means that there is a large difference between the current shot and the shot of its neighboring shot. In this case, the two shots do not belong to the same scene and scene segmentation is required.

[0121] 409. If the difference between the current shot and its adjacent shot is not greater than the preset value, then the current shot and its adjacent shot are determined to be shots of the same scene.

[0122] When the difference value is less than the preset value, it means that the difference between the current shot and its adjacent shot is small, and the two shots belong to the same scene.

[0123] By performing the above operation on each shot in sequence, the differences between each shot and its adjacent shots are determined, thereby determining the segmentation position.

[0124] For example, such as Figure 4c As shown, based on Figure 1b The video shown is divided into shot sequences by dividing the shot frames. Then, multimodal feature information is extracted from each shot frame and fused separately. In addition, difference calculation is performed to achieve scene division in sequence.

[0125] This solution extracts the differences between adjacent shots and uses the magnitude of these differences as a key indicator for determining whether a scene should be segmented. This approach can improve the accuracy of scene segmentation.

[0126] This application embodiment segments a video into multiple shot frames, performs multimodal feature extraction, and fuses the multimodal features of each shot frame. Based on the fused semantic information of the shot frames, the difference values ​​between adjacent shots are determined, thereby identifying the scene segmentation location and achieving scene segmentation of the video. This solution combines the advantages of unsupervised algorithms, which extract discriminative features of shots without relying on labeled samples, with the advantages of supervised algorithms, utilizing pre-trained models of deep learning algorithms to extract multimodal features and determine the scene segmentation location based on the fused semantic information of the shot frames. Furthermore, this solution extracts the differences between adjacent shots and uses the magnitude of these differences as a key indicator for determining the scene segmentation location, thus improving the accuracy of scene segmentation.

[0127] Based on the description of the above-described video scene segmentation method embodiments, this invention also discloses a video scene segmentation apparatus, with reference to... Figure 5 , Figure 5 This is a schematic diagram of a video scene segmentation device provided in an embodiment of the present invention. The device includes a segmentation module 501, an extraction module 502, a fusion module 503, and a segmentation module 504; wherein:

[0128] The video segmentation module 501 is used to segment the video to obtain multiple shot frames;

[0129] The extraction module 502 is used to extract features from the multiple shot images respectively to obtain the multimodal features of each shot image in the multiple shot images;

[0130] The fusion module 503 is used to perform fusion processing on the multimodal features of each shot in the multiple shot images to obtain the fused semantic information of each shot in the multiple shot images.

[0131] The segmentation module 504 is used to determine the scene segmentation position in the multiple shot images based on the fusion semantic information of each shot image in the multiple shot images, and to perform scene segmentation on the multiple shot images.

[0132] The segmentation module 504 is used for:

[0133] The difference value between each shot and its adjacent shot is determined based on the fusion semantic information of each shot in the plurality of shot frames;

[0134] Based on the difference between each shot and its adjacent shot, the scene segmentation position in the plurality of shot images is determined, and the plurality of shot images are segmented.

[0135] Furthermore, the segmentation module 504 is also used to: determine a first group of frames and a second group of frames based on the current shot image, wherein the current shot image and the K consecutive shots preceding it are determined as the first group of frames, and the K consecutive shots following the current shot image are determined as the second group of frames.

[0136] The fusion semantic information of the shots contained in the first group of shots and the fusion semantic information of the shots contained in the second group of shots are respectively convolved to obtain a first fusion semantic vector and a second fusion semantic vector. The first fusion semantic vector corresponds to the fusion semantic information of the shots contained in the first group of shots, and the second fusion semantic vector corresponds to the fusion semantic information of the shots contained in the second group of shots.

[0137] The difference value between the current shot and its adjacent shot is determined based on the first fusion semantic vector and the second fusion semantic vector, wherein the difference value between each of the plurality of shot images and its adjacent shot images includes the difference value between the current shot and its adjacent shot images.

[0138] Furthermore, the segmentation module 504 is used to: calculate the difference distance based on the first fused semantic vector and the second fused semantic vector;

[0139] The difference value between the current shot and its adjacent shot is determined based on the difference distance.

[0140] If the difference between the current shot and its adjacent shot is greater than a preset value, then the position between the current shot and its adjacent shot is determined as the segmentation position.

[0141] If the difference between the current shot and its adjacent shot is not greater than the preset value, then the current shot and its adjacent shot are determined to be shots of the same scene.

[0142] Optionally, the multimodal features of each shot include at least two of the following: speech features, text features, and image features.

[0143] This application's embodiments segment the video into multiple shot frames, then perform multimodal feature extraction, and fuse the multimodal features of each shot frame separately. Based on the fused semantic information of the shot frames, the scene segmentation location is determined, thus achieving video scene segmentation. This scheme combines the advantages of unsupervised algorithms, which extract discriminative features of shots without relying on labeled samples, with the advantages of supervised algorithms, utilizing pre-trained models of deep learning algorithms to extract multimodal features. These features are then fused online to fully represent the shots, and the scene segmentation location is determined based on the fused semantic information of the shot frames. This approach improves the accuracy of video scene segmentation.

[0144] It is worth noting that the specific functional implementation of the video scene segmentation device can be found in the description of the video scene segmentation method above, and will not be repeated here. The various units or modules in the video scene segmentation device can be individually or entirely merged into one or more other units or modules, or some of the units or modules can be further divided into multiple functionally smaller units or modules. This achieves the same operation without affecting the technical effect of the embodiments of the present invention. The aforementioned units or modules are based on logical function division. In practical applications, the function of one unit (or module) can also be implemented by multiple units (or modules), or the function of multiple units (or modules) can be implemented by one unit (or module).

[0145] Based on the description of the above method and apparatus embodiments, this invention also provides an apparatus for video scene segmentation. For example... Figure 6 The device 600 shown (which may specifically be a computer device) includes a memory 601, a processor 602, a communication interface 603, and a bus 604. The memory 601, processor 602, and communication interface 603 are interconnected via the bus 604.

[0146] The memory 601 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).

[0147] The memory 601 can store a program. When the program stored in the memory 601 is executed by the processor 602, the processor 602 and the communication interface 603 are used to execute the various steps of the video scene segmentation method of the present application embodiment.

[0148] The processor 602 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the video scene segmentation apparatus of this application embodiment, or to execute the video scene segmentation method of this application method embodiment.

[0149] The processor 602 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the video scene segmentation method of this application can be completed by the integrated logic circuits in the hardware of the processor 602 or by instructions in software form. The aforementioned processor 602 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 601. The processor 602 reads the information in the memory 601 and, in conjunction with its hardware, performs the functions required by the units included in the video scene segmentation apparatus of this application embodiment, or performs the video scene segmentation method of this application method embodiment.

[0150] The communication interface 603 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 600 and other devices or communication networks. For example, data can be acquired through the communication interface 603.

[0151] Bus 604 may include a pathway for transmitting information between various components of device 600 (e.g., memory 601, processor 602, communication interface 603).

[0152] It should be noted that, although Figure 6 The illustrated device 600 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, device 600 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that device 600 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that device 600 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 6 All the devices shown.

[0153] This application also provides a chip system applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the video scene segmentation method.

[0154] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of any of the above methods.

[0155] This application also provides a computer program product containing instructions. When the computer program product is run on a computer or processor, it causes the computer or processor to perform one or more steps of any of the methods described above.

[0156] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the specific descriptions of the corresponding steps in the foregoing method embodiments, and will not be repeated here.

[0157] It should be understood that, in the description of this application, unless otherwise stated, " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B can represent A or B; where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Additionally, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply difference. In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.

[0158] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling, direct coupling, or communication connection shown or discussed between each other may be indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be read-only memory (ROM), random access memory (RAM), or magnetic media, such as floppy disks, hard disks, magnetic tapes, magnetic disks, or optical media, such as digital versatile discs (DVDs), or semiconductor media, such as solid-state disks (SSDs).

[0161] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A method for video scene segmentation, characterized in that, include: The video is segmented to obtain multiple shot frames; Feature extraction is performed on each of the multiple shot frames to obtain the multimodal features of each shot frame in the multiple shot frames; The multimodal features of each shot in the multiple shot frames are fused separately to obtain the fused semantic information of each shot in the multiple shot frames; The current shot and the K consecutive shots preceding it are defined as the first group of shots, and the K consecutive shots following the current shot are defined as the second group of shots, wherein the current shot is any shot among the plurality of shots. The fusion semantic information of the shots contained in the first group of shots and the fusion semantic information of the shots contained in the second group of shots are respectively convolved to obtain a first fusion semantic vector and a second fusion semantic vector. The first fusion semantic vector corresponds to the fusion semantic information of the shots contained in the first group of shots, and the second fusion semantic vector corresponds to the fusion semantic information of the shots contained in the second group of shots. The difference value between the current shot and its adjacent shot is determined based on the first fusion semantic vector and the second fusion semantic vector, wherein the difference value between the current shot and its adjacent shot is any one of the difference values ​​between each shot and its adjacent shot in the plurality of shot shots; Based on the difference between each shot and its adjacent shots, the scene segmentation position in the multiple shot frames is determined, and the multiple shot frames are segmented.

2. The method according to claim 1, characterized in that, Determining the difference value between the current shot and its adjacent shot based on the first fused semantic vector and the second fused semantic vector includes: The difference distance is calculated based on the first fused semantic vector and the second fused semantic vector; The difference value between the current shot and its adjacent shot is determined based on the difference distance.

3. The method according to claim 1 or 2, characterized in that, Determining the scene segmentation position among the multiple shot frames based on the difference value between each shot frame and its adjacent shot frames includes: If the difference between the current shot and its adjacent shot is greater than a preset value, then the position between the current shot and its adjacent shot is determined as the segmentation position. If the difference between the current shot and its adjacent shot is not greater than the preset value, then the current shot and its adjacent shot are determined to be shots of the same scene.

4. The method according to claim 1, characterized in that, The multimodal features of each shot include at least two of the following: speech features, text features, and image features.

5. A device for video scene segmentation, characterized in that, include: The video segmentation module is used to segment the video to obtain multiple shot frames; The extraction module is used to extract features from the multiple shot images to obtain the multimodal features of each shot image in the multiple shot images; The fusion module is used to perform fusion processing on the multimodal features of each shot in the multiple shot images to obtain the fused semantic information of each shot in the multiple shot images; The segmentation module is used to determine the scene segmentation position in the multiple shot frames based on the fusion semantic information of each shot frame in the multiple shot frames, and to perform scene segmentation on the multiple shot frames; The segmentation module, when determining the scene segmentation position in the multiple shot frames based on the fused semantic information of each shot frame, is further configured to: The current shot and the K consecutive shots preceding it are defined as the first group of shots, and the K consecutive shots following the current shot are defined as the second group of shots, wherein the current shot is any shot among the plurality of shots. The fusion semantic information of the shots contained in the first group of shots and the fusion semantic information of the shots contained in the second group of shots are respectively convolved to obtain a first fusion semantic vector and a second fusion semantic vector. The first fusion semantic vector corresponds to the fusion semantic information of the shots contained in the first group of shots, and the second fusion semantic vector corresponds to the fusion semantic information of the shots contained in the second group of shots. The difference value between the current shot and its adjacent shot is determined based on the first fusion semantic vector and the second fusion semantic vector, wherein the difference value between the current shot and its adjacent shot is any one of the difference values ​​between each shot and its adjacent shot in the plurality of shot shots; The scene segmentation position in the multiple shot frames is determined based on the difference value between each shot frame and its adjacent shot frames.

6. A computer program product comprising a computer-readable storage medium storing a computer program, characterized in that, When executed by at least one processor, the computer program causes the at least one processor to perform the video scene segmentation method as described in any one of claims 1-4.

7. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which includes program instructions that, when executed by a processor, perform the video scene segmentation method as described in any one of claims 1-4.

Images