Video segmentation method, device, equipment and storage medium

Abstract

The present disclosure relates to a video segmentation method, device, equipment and storage medium, and relates to the field of image processing. The embodiments of the present disclosure at least solve the problem of inaccurate video segmentation based on scenes in the related art. The method includes: obtaining a video to be segmented, and determining a plurality of video segments included in the video to be segmented; determining the segment features of each video segment; dividing the plurality of video segments into a plurality of scenes based on the segment features of each video segment and a pre-trained self-attention model; the self-attention model is used to determine the correlation between two adjacent video segments according to the segment features of each video segment in the two adjacent video segments; and segmenting the video to be segmented based on the plurality of scenes obtained by the division.

Description

Technical Field

[0001] The present disclosure relates to the computer field, and in particular to a video segmentation method, device, equipment and storage medium. Background Art

[0002] In the task of video processing, a video segmentation method based on a convolutional neural network can be used to segment the video into multiple scenes according to the different scenes of the video shooting. Each scene is a video clip composed of many different images; at the same time, there is a high degree of similarity and semantic continuity between the multiple image frames included in each video clip. Specifically, a pre-trained convolutional neural network is used to extract the image features of each frame in the video, and based on the similarity between the image features, it is determined whether two adjacent image frames belong to the same scene, and the video is further segmented based on the judgment result.

[0003] However, the above-mentioned video segmentation method is naturally suitable for extracting image features due to the inductive bias characteristics of convolutional neural networks, but the performance of comparing and summarizing the features of adjacent image frames is insufficient, and the results of scene-based video segmentation are not accurate enough. Summary of the invention

[0004] The present disclosure provides a video segmentation method, apparatus, device and storage medium to at least solve the problem of inaccurate video segmentation based on scenes in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a video segmentation method is provided, comprising: obtaining a video to be segmented, and determining a plurality of video segments included in the video to be segmented; determining segment features of each video segment; dividing the plurality of video segments into a plurality of scenes based on the segment features of each video segment and a pre-trained self-attention model; the self-attention model is used to determine the correlation between two adjacent video segments according to the segment features of each video segment in the two adjacent video segments; and segmenting the video to be segmented based on the plurality of scenes obtained by the division.

[0006] Optionally, the above-mentioned determining the segment features of each video segment includes: determining image features of each video segment and text features of each video segment; and determining the segment features of each video segment according to the image features and text features of each video segment.

[0007] Optionally, the above-mentioned determination of the image features of each video clip and the text features of each video clip includes: determining a key image frame from multiple image frames included in each video clip; the key image frame is located at a preset position in the multiple image frames; based on the key image frame and a pre-trained first neural network, the image features of the key image frame are determined, and the image features of the key image frame are determined as the image features of each video clip; the first neural network is used to perform convolution processing on the input image frame to obtain corresponding image features; determine text information of the multiple image frames, and determine the text features of each video clip based on the text information and a pre-trained second neural network; the second neural network is used to perform convolution processing on the input text information to obtain corresponding text features.

[0008] Optionally, the above-mentioned determining the image features of the key image frame based on the key image frame and the pre-trained first neural network includes: cropping the key image frame into multiple sub-image frames, and generating image features of the multiple sub-image frames based on the multiple sub-image frames and the first neural network; and averaging the image features of the multiple sub-image frames to obtain the image features of the key image frame.

[0009] Optionally, the above-mentioned method of dividing multiple video clips into multiple scenes based on the clip features of each video clip and the pre-trained self-attention model includes: inputting the image features of each video clip and the text features of each video clip into the first self-attention layer of the self-attention model, and weighting the image features of each video clip and the text features of each video clip based on the feature weights determined by the first self-attention layer to obtain a first fusion feature of each video clip; for adjacent video clips in the multiple video clips, inputting the first fusion features of the adjacent video clips into the second self-attention layer of the self-attention model, and weighting the first fusion features of the adjacent video clips based on the feature weights determined by the second self-attention layer to obtain second fusion features of the corresponding adjacent video clips; inputting the second fusion features of the corresponding adjacent video clips into the fully connected layer of the self-attention model to determine the correlation between the adjacent video clips; determining the scene segmentation results for the adjacent video clips based on the correlation between the adjacent video clips; the scene segmentation results indicate whether the adjacent video clips are located in the same scene; and dividing the multiple video clips into multiple scenes based on the scene segmentation results of each adjacent video clip in the multiple video clips.

[0010] According to a second aspect of an embodiment of the present disclosure, a video segmentation device is provided, including an acquisition unit, a determination unit and a processing unit; the acquisition unit is used to acquire a video to be segmented; the determination unit is used to determine a plurality of video segments included in the video to be segmented; the determination unit is also used to determine a segment feature of each video segment; the processing unit is used to divide the plurality of video segments into a plurality of scenes based on the segment feature of each video segment and a pre-trained self-attention model; the self-attention model is used to determine the correlation between two adjacent video segments according to the segment feature of each video segment in the two adjacent video segments; the processing unit is also used to segment the video to be segmented based on the plurality of scenes obtained by the division.

[0011] Optionally, the above-mentioned determination unit is specifically used to: determine the image features of each video segment and the text features of each video segment; determine the segment features of each video segment according to the image features and the text features of each video segment.

[0012] Optionally, the above-mentioned determination unit is specifically used to: determine a key image frame from multiple image frames included in each video clip; the key image frame is located at a preset position in the multiple image frames; based on the key image frame and a pre-trained first neural network, determine the image features of the key image frame, and determine the image features of the key image frame as the image features of each video clip; the first neural network is used to perform convolution processing on the input image frame to obtain corresponding image features; determine text information of multiple image frames, and determine text features of each video clip based on the text information and a pre-trained second neural network; the second neural network is used to perform convolution processing on the input text information to obtain corresponding text features.

[0013] Optionally, the above-mentioned determination unit is specifically used to: crop the key image frame into multiple sub-image frames, and generate image features of the multiple sub-image frames based on the multiple sub-image frames and the first neural network; average pool the image features of the multiple sub-image frames to obtain the image features of the key image frame.

[0014] Optionally, the above-mentioned processing unit is specifically used to: input the image features of each video clip and the text features of each video clip into the first self-attention layer of the self-attention model, and weight the image features of each video clip and the text features of each video clip based on the feature weights determined by the first self-attention layer to obtain the first fusion features of each video clip; for adjacent video clips among multiple video clips, input the first fusion features of the adjacent video clips into the second self-attention layer of the self-attention model, and weight the first fusion features of the adjacent video clips based on the feature weights determined by the second self-attention layer to obtain the second fusion features of the corresponding adjacent video clips; input the second fusion features of the corresponding adjacent video clips into the fully connected layer of the self-attention model to determine the correlation between adjacent video clips; determine the scene segmentation results for the adjacent video clips based on the correlation between the adjacent video clips; the scene segmentation results indicate whether the adjacent video clips are located in the same scene; and divide the multiple video clips into multiple scenes based on the scene segmentation results of each adjacent video clip among the multiple video clips.

[0015] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the video segmentation method provided in the first aspect.

[0016] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium, comprising instructions, which, when executed by a processor of an electronic device, enable the electronic device to perform the video segmentation method provided in the first aspect.

[0017] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor of an electronic device, enable the electronic device to perform the video segmentation method provided in the first aspect.

[0018] The technical solution provided by the above embodiment has at least the following beneficial effects: the self-attention model can fully exploit the characteristics of long-range dependencies between features, and judge whether there is a correlation between the features of any two adjacent video clips, so as to determine whether the two adjacent video clips are in the same scene. In this way, the multiple scenes where the video clips included in the video to be segmented are located can be determined, and the accuracy of the result of segmenting the video to be segmented can be achieved.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0021] Figure 1 is a structural schematic diagram of a multi-video segmentation system according to an exemplary embodiment;

[0022] Figure 2 is one of the flowchart diagrams of a video segmentation method according to an exemplary embodiment;

[0023] Figure 3 is a second flowchart diagram of a video segmentation method according to an exemplary embodiment;

[0024] Figure 4 is a third flowchart of a video segmentation method according to an exemplary embodiment;

[0025] Figure 5 is a fourth flowchart diagram of a video segmentation method according to an exemplary embodiment;

[0026] Figure 6 is a fifth flowchart of a video segmentation method according to an exemplary embodiment;

[0027] Figure 7 is a structural schematic diagram of a video segmentation device according to an exemplary embodiment;

[0028] Figure 8 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0031] In addition, in the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or, for example, A / B can mean A or B. "And / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present disclosure, "multiple" means two or more than two.

[0032] The video segmentation method provided by the embodiment of the present disclosure may be applicable to a video segmentation system. Figure 1 FIG. 4 shows a structural diagram of the video segmentation system. Figure 1 As shown, the video segmentation system 10 is used to segment a video based on a scene and improve the accuracy of the video segmentation. The video segmentation system 10 includes a video segmentation device 11 and an electronic device 12. The video segmentation device 11 is connected to the electronic device 12. The video segmentation device 11 and the electronic device 12 can be connected in a wired manner or in a wireless manner, which is not limited in the embodiment of the present disclosure.

[0033] The video segmentation device 11 can be used to receive a video sent by the electronic device 12. For example, the video segmentation device 11 can be used to receive a video to be segmented by the electronic device 12, and cut the video to be segmented into multiple video segments. Further, the video segmentation device can also divide the multiple video segments into multiple scenes, and determine the scene segmentation points between the scenes according to the scenes obtained by the division, and send the determined scene segmentation points to the electronic device.

[0034] The electronic device 12 may be used to receive the scene segmentation points sent by the video segmentation device 11, and segment the video to be segmented according to the received scene segmentation points.

[0035] The electronic device 12 in the video segmentation system 10 involved in the embodiment of the present disclosure may be a server for segmenting a video, or a server cluster composed of multiple servers, or a cloud computing service center.

[0036] It should be noted that the video segmentation device 11 and the electronic device 12 may be independent devices or integrated into the same device, and this disclosure does not specifically limit this.

[0037] When the video segmentation device 11 and the electronic device 12 are integrated into the same device, the communication between the video segmentation device 11 and the electronic device 12 is the communication between the internal modules of the device. In this case, the communication process between the two is the same as "the communication process between the video segmentation device 11 and the electronic device 12 when they are independent of each other".

[0038] In the following embodiments provided by the present disclosure, the present disclosure is described by taking the video segmentation device 11 and the electronic device 12 as being independently configured. At the same time, since the video segmentation method provided by the embodiments of the present disclosure can be applied to the above-mentioned video segmentation device, and can also be applied to an electronic device or other similar device including the video segmentation device, the embodiments of the present disclosure are combined with the accompanying drawings, taking the application to the video segmentation device as an example, to describe the video segmentation method provided by the embodiments of the present disclosure.

[0039] like Figure 2 As shown, the video segmentation method provided by the embodiment of the present disclosure includes the following S201-S205.

[0040] S201: The video segmentation device obtains a video to be segmented.

[0041] As a possible implementation, after the user issues an operation instruction to the electronic device to instruct to segment the video to be segmented, the electronic device sends the video to be segmented to the video segmentation device in response to the received operation instruction. Correspondingly, the video segmentation device receives the video to be segmented.

[0042] It should be noted that the video to be segmented may be a video pre-stored in a multimedia resource library of the electronic device, or may be a video uploaded by a user.

[0043] S202: The video segmentation apparatus determines a plurality of video segments included in the video to be segmented.

[0044] As a possible implementation manner, the video segmentation device segments the video to be segmented into multiple video segments according to preset durations.

[0045] In some embodiments, each video segment in the plurality of video segments has the same duration.

[0046] Exemplarily, the duration of each video clip may be 0.1 second (s).

[0047] S203: The video segmentation device determines the segment features of each video segment.

[0048] The segment features are used to identify the image frames included in the video segment and the text information included in the video segment.

[0049] As a possible implementation manner, the video segmentation device may merge the image features of the image frames included in each video segment to obtain the image features of each video segment.

[0050] At the same time, the video segmentation device can also merge text information such as subtitles and description information in the image frames included in each video segment, and determine the text features of the video segment according to the merged text information.

[0051] Furthermore, the video segmentation device fuses the determined image features and text features to obtain segment features of each video segment.

[0052] For the specific implementation method of this step, please refer to the subsequent description of the embodiment of the present disclosure, which will not be repeated here.

[0053] S204, the video segmentation device divides the multiple video segments into multiple scenes based on the segment features of each video segment and the pre-trained self-attention model.

[0054] Among them, the self-attention model is used to determine the correlation between two adjacent video clips based on the clip features of the two adjacent video clips.

[0055] As a possible implementation method, the video segmentation device sequentially inputs the segment features of two adjacent video segments into the self-attention model, and determines the correlation between the segment features of the two video segments based on the output results of the self-attention model, and then determines whether the two video segments are the same scene, until all two adjacent video segments are the same scene.

[0056] It can be understood that by adopting this step, multiple video clips can be divided into multiple scenes.

[0057] It should be noted that the above self-attention model is a pre-trained model, which includes multiple self-attention layers and a fully connected layer. Each self-attention layer is used to calculate the weighted fusion features between multiple input features, and the fully connected layer is used to calculate the correlation between multiple input features.

[0058] The specific implementation of this step can refer to the subsequent description of the embodiment of the present disclosure, which will not be repeated here. In addition, in this step, the specific description of the self-attention model can refer to the subsequent description of the embodiment of the present disclosure or the prior art, which will not be repeated here.

[0059] S205: The video segmentation device segments the video to be segmented based on the multiple scenes obtained by segmentation.

[0060] As a possible implementation, after determining that the multiple video clips are located in multiple scenes, the video segmentation device determines a scene segmentation point between two adjacent scenes. Further, the video segmentation device sends the determined scene segmentation point to the electronic device, so that the electronic device segments the video to be segmented based on the received scene segmentation point.

[0061] It should be noted that the above-mentioned scene division points are the switching moments between adjacent video clips in different scenes.

[0062] Exemplarily, video segment a is adjacent to video segment b, and video segment a is located in scene 1, and video segment b is located in scene 2. In this case, the scene division point between scene 1 and scene 2 is the end time of video segment a, or the start time of video segment b.

[0063] The technical solution provided by the above embodiment has at least the following beneficial effects: the self-attention model can fully exploit the characteristics of long-range dependencies between features, and judge whether there is a correlation between the features of any two adjacent video clips, so as to determine whether the two adjacent video clips are in the same scene. In this way, the multiple scenes where the video clips included in the video to be segmented are located can be determined, and the accuracy of the result of segmenting the video to be segmented can be achieved.

[0064] In one design, in order to determine the segment features of each video segment and reduce the computing resources of the device, such as Figure 3 As shown, S203 provided in the embodiment of the present disclosure specifically includes the following S2031-S2032.

[0065] S2031. The video segmentation device determines image features of each video segment and text features of each video segment.

[0066] The image feature of each video clip is used to identify the image frame included in each video clip, and the text feature of each video clip is used to identify the text information included in each video clip.

[0067] As a possible implementation, the video segmentation device can obtain the image frames included in each video clip, and determine the image features of the multiple image frames based on the multiple image frames included in each video clip and a pre-trained image processing neural network. Further, the video segmentation device merges the determined image features of the multiple image frames, and uses the merged image features as the image features of the video clip.

[0068] On the other hand, after acquiring the image frames included in each video clip, the video segmentation device also acquires the text information in each image frame, and determines the text features corresponding to each image frame based on the acquired text information and a preset text processing neural network. Further, the video segmentation device merges the determined multiple text features, and uses the merged question feature as the text feature of the video clip.

[0069] The specific implementation method of this step can also refer to the subsequent description of the embodiment of the present disclosure, which will not be repeated here.

[0070] S2032: The video segmentation device determines the segment features of each video segment according to the image features and text features of each video segment.

[0071] As a possible implementation manner, the video segmentation device may merge the image features and text features of each video segment to obtain the segment features of each video segment.

[0072] As another possible implementation manner, the video segmentation device may also determine the segment features of each video segment by determining the image features of each video segment and the text features of each video segment as the segment features of each video segment.

[0073] The technical solution provided by the above embodiment has at least the following beneficial effects: determining the segment features of each video segment based on the obtained image features and text features of the video segment, which can truly reflect the multiple image frames included in the video segment and the text information in the multiple image frames, and thus can achieve the accuracy of the results of segmenting the video to be segmented.

[0074] In one design, in order to determine the image features of each video segment and the text features of each video segment, such as Figure 4 As shown, S2031 provided in the embodiment of the present disclosure specifically includes the following S301-S305.

[0075] S301: A video segmentation apparatus determines a key image frame from a plurality of image frames included in each video segment.

[0076] The key image frame is located at a preset position among the multiple image frames.

[0077] As a possible implementation manner, the video segmentation device determines a key image frame from a plurality of image frames based on a preset position that is set in advance.

[0078] It should be noted that the preset position may be the middle position of a plurality of image frames. The preset position may also be set in advance in the video segmentation device or electronic device by the operation and maintenance personnel of the video segmentation system.

[0079] Exemplarily, taking a video clip including 6 image frames as an example, the key image frame may be the 3rd or 4th of the 6 image frames.

[0080] It can be understood that by using the image frame at the middle position as the key image frame, the key image frame can be used to more accurately represent the multiple image frames included in the video clip.

[0081] S302: The video segmentation device determines image features of the key image frame based on the key image frame and a pre-trained first neural network.

[0082] The first neural network is used to perform convolution processing on the input image frame to obtain corresponding image features.

[0083] As a possible implementation manner, the video segmentation device inputs the key image frame into the first neural network so that the first neural network outputs image features of the key image frame.

[0084] Exemplarily, the first neural network may be a resnet50 network.

[0085] S303: The video segmentation device determines the image features of the key image frame as the image features of each video segment.

[0086] S304: The video segmentation device determines text information of multiple image frames.

[0087] As a possible implementation manner, the video segmentation device processes a plurality of image frames included in each video frequency band based on a preset optical character recognition (OCR) model to obtain text information in the plurality of image frames.

[0088] It should be noted that the above text information may be subtitles in multiple image frames, may be description information in multiple image frames, or may be scene information in multiple image frames.

[0089] S305: The video segmentation device determines the text features of each video clip according to the text information and the pre-trained second neural network.

[0090] Among them, the second neural network is used to perform convolution processing on the input text information to obtain corresponding text features.

[0091] As a possible implementation method, the video segmentation device inputs the text information in the acquired multiple image frames into the second neural network, and uses the result output by the second neural network as the text feature of the video clip.

[0092] Exemplarily, the second neural network can be the encoder part in the BERT model.

[0093] It should be noted that, when the second neural network is the encoder part, the text feature of the video clip can be a 768-dimensional text feature.

[0094] The technical solution provided by the above embodiment has at least the following beneficial effects: using the image features of key image frames to reflect the image features of image frames included in the video clip, and using the text features of multiple image frames to reflect the text features of text information in the image frames included in the video clip, which can greatly reduce the computing resources of the video segmentation device or electronic device and improve the processing speed.

[0095] In one design, in order to ensure that the first neural network can fully learn the content in the key image frame, such as Figure 5 As shown, S302 provided in the embodiment of the present disclosure may specifically include S3021-S3023.

[0096] S3021. The video segmentation device cuts the key image frame into multiple sub-image frames.

[0097] The size of each sub-image frame is the same.

[0098] As a possible implementation manner, the video segmentation device cuts the key image frame into multiple sub-image frames based on a preset size.

[0099] In some embodiments, the video segmentation device may crop the key image frame according to a preset size using a cropping method with a sliding step size.

[0100] Exemplarily, the size of each sub-image frame may be 224*224.

[0101] S3022. The video segmentation device generates image features of multiple sub-image frames according to the multiple sub-image frames and the first neural network.

[0102] As a possible implementation manner, the video segmentation device inputs multiple sub-image frames into the first neural network to obtain image features of the multiple sub-image frames output by the first neural network.

[0103] As another possible implementation, the video segmentation device performs a data enhancement operation on each of the multiple sub-image frames to obtain multiple enhanced sub-image frames. Further, the video segmentation device transforms the multiple enhanced sub-image frames into tensor data, and inputs the transformed tensor data into the first neural network to obtain image features of the multiple sub-image frames output by the first neural network.

[0104] It should be noted that the above enhancement operation may be a cutout enhancement method. At the same time, the video segmentation device may use pytouch to transform the multiple enhanced sub-image frames into tensor data with a dimension of [b, 3, 224, 224].

[0105] Wherein, b is the number of the plurality of sub-image frames, 3 is the dimension of each sub-image frame, and 224 is the size of each sub-image frame.

[0106] Exemplarily, the first neural network includes four different convolutional layers layer1, layer2, layer3 and layer4, each of which includes multiple blocks, and each block consists of three layers of convolution. The tensor data of dimension [b, 3, 224, 224] input to the first neural network passes through the above four convolutional layers in sequence, and finally outputs tensor data of dimension [b, 2048, 7, 7]. Thus, the video segmentation device uses the output tensor data as the image feature of each sub-image frame.

[0107] Below, a training method for the first neural network is also introduced:

[0108] In the sample process of the training process, the video segmentation device cuts the acquired sample image frame into multiple sub-sample image frames. Further, the video segmentation device performs enhancement processing on each sub-sample image frame based on the above-mentioned cutout enhancement method to obtain multiple enhanced sub-sample image frames, and transforms the obtained multiple enhanced sub-sample image frames into sample tensor data. Thus, the video segmentation device can use the sample tensor data as a sample to train the first neural network.

[0109] It should be noted that the process of obtaining sample data in this step can refer to the specific description of the second implementation method in S3022 above, and will not be repeated here.

[0110] S3023: The video segmentation device performs average pooling on the image features of the multiple sub-image frames to obtain image features of the key image frame.

[0111] As a possible implementation manner, the video segmentation device performs an average pooling operation on the image features of the plurality of sub-image frames, and uses the processed image features as the image features of the key image frame.

[0112] In this step, the implementation method of the average pooling operation can refer to the prior art and will not be described in detail here.

[0113] The technical solution provided by the above embodiment has at least the following beneficial effects: the key image frame is cut into multiple sub-image frames, and the image features of the key image frame are determined using the multiple sub-image frames and the first neural network. Since the first neural network can learn more features from the multiple sub-image frames, the image features of the key image frame determined are more accurate and sufficient, thereby improving the accuracy of the determined image features.

[0114] In one design, in order to be able to divide multiple video clips into multiple scenes, such as Figure 6 As shown, S204 provided in the embodiment of the present disclosure specifically includes the following S2041-S2047.

[0115] S2041. The video segmentation device inputs the image features of each video clip and the text features of each video clip into the first self-attention layer of the self-attention model to obtain the feature weights of the image features of each video clip and the feature weights of the text features of each video clip.

[0116] Exemplarily, taking the image feature a and the text feature b input into the first self-attention layer as an example, the video segmentation device inputs the image feature a and the text feature b of each video clip into the first self-attention layer, and the first self-attention layer learns the characteristic image feature a and the text feature b to obtain the feature weight α of the image feature a and the feature weight β of the text feature b.

[0117] It can be understood that the feature weights output by the first self-attention layer can characterize the correlation and dependency between the input image features and text features.

[0118] The following is a description of the process of determining feature weights in the first self-attention layer in combination with the formula in the first self-attention layer. Taking the feature weights of image features as an example, the feature weights of the image features of each video clip satisfy the following formula:

[0119]

[0120] Among them, S is the feature weight of the image feature of each video clip, Q is the query transformation vector of the image feature of the above video clip, and K T is the transpose of the key conversion vector of the image features of the above video clip, V is the value conversion vector of the image features of the above video clip, and d is the number of features input in the first self-attention layer.

[0121] The method for determining the text features of the video clip can also refer to the above formula, which will not be described in detail here.

[0122] S2042. The video segmentation device weights the image features of each video clip and the text features of each video clip based on the feature weights determined by the first self-attention layer to obtain a first fusion feature of each video clip.

[0123] As a possible implementation method, the video segmentation device weights the image features of each video clip and the text features of each video clip according to the feature weights of the image features of each video clip and the feature weights of the text features of each video clip to obtain a first fusion feature of each video clip.

[0124] Exemplarily, taking the above example, the video segmentation device uses the sum of the product of the image feature a and its feature weight α and the product of the text feature b and its feature weight β as the first fusion feature of the video clip.

[0125] It can be understood that the first fusion feature can fuse the image feature of each video clip and the text feature, and the first fusion feature can reflect the correlation between the image feature and the text feature of each video clip.

[0126] S2043. The video segmentation device inputs the first fusion features of adjacent video segments in the multiple video segments into the second self-attention layer of the self-attention model to obtain the feature weights of the first video segment and the feature weights of the second video segment in the adjacent video segments.

[0127] As a possible implementation method, the video segmentation device inputs the first fusion feature of the first video frequency band and the first fusion feature of the second video frequency band in adjacent video segments into the second self-attention layer, and the second self-attention layer learns the first fusion feature of the first video frequency band and the first fusion feature of the second video frequency band to obtain the feature weight α of the first fusion feature of the first video frequency band and the feature weight β of the first fusion feature of the second video frequency band.

[0128] It can be understood that the feature weights output by the second self-attention layer can characterize the correlation and dependency between the first fused features of the input first video frequency band and the first fused features of the second video frequency band.

[0129] The method for determining the feature weight of the first fusion feature of each video clip can also refer to the above formula, which will not be described in detail here.

[0130] S2044. The video segmentation device weights the first fusion features of adjacent video clips based on the feature weights determined by the second self-attention layer to obtain second fusion features corresponding to the adjacent video clips.

[0131] As a possible implementation method, the video segmentation device weights the first fusion feature of the first video frequency band in the adjacent video segment and the first fusion feature of the second video frequency band in the adjacent video segment according to the feature weight of the first fusion feature of the first video frequency band in the adjacent video segment and the feature weight of the first fusion feature of the second video frequency band to obtain the second fusion feature of the adjacent video segment.

[0132] It can be understood that the second fusion feature can fuse the first fusion feature of the first video frequency band in the adjacent video segments and the first fusion feature of the second video frequency band, and the second fusion feature can reflect the correlation between the first fusion features of the adjacent video segments.

[0133] In some embodiments, the process of obtaining the second fused feature here can be achieved by connecting a feature fusion layer after the second self-attention layer in the self-attention model. The feature fusion layer can be used to perform a cls-token operation, and fuse the first fused features of adjacent video clips through the cls-token operation to obtain the second fused feature.

[0134] In some embodiments, the second fused feature of any adjacent video segments may be a 1024-dimensional feature.

[0135] S2045. The video segmentation device inputs the second fusion features corresponding to the adjacent video segments into the fully connected layer of the self-attention model to determine the correlation between the adjacent video segments.

[0136] As a possible implementation method, the video segmentation device inputs the second fusion features of two adjacent video clips into the FC fully connected layer of the self-attention model. After the fully connected layer is subjected to binary classification, the correlation between the adjacent video clips is obtained.

[0137] Exemplarily, the video segmentation device inputs the second fusion features corresponding to the adjacent video segments into the fully connected layer of the self-attention model, and the fully connected layer can output a classification result based on the binary classification, and the classification result is used to reflect the correlation between the adjacent video segments. When the classification result is 0, it indicates that the adjacent segments are not correlated, and when the classification result is 1, it indicates that the adjacent segments are correlated.

[0138] S2046: The video segmentation device determines scene segmentation results for adjacent video segments according to the correlation between adjacent video segments.

[0139] The scene segmentation result indicates whether adjacent video clips are located in the same scene.

[0140] As a possible implementation manner, the video segmentation device determines the correlation between adjacent video segments based on the classification results output by the fully connected layer, and determines whether the adjacent video segments are located in the same scene based on the determined correlation.

[0141] Exemplarily, if the classification result output by the fully connected layer is 0, it indicates that the two adjacent video clips are not located in the same scene; if the classification result output by the fully connected layer is 1, it indicates that the two adjacent clips are located in the same scene.

[0142] S2047: The video segmentation device divides the multiple video segments into multiple scenes according to the scene segmentation result of each adjacent video segment in the multiple video segments.

[0143] As a possible implementation manner, the video segmentation device repeatedly executes the above S2041-S2046 for each adjacent video segment until the scene segmentation result of each adjacent video segment in the multiple video segments is determined, and then the multiple video segments included in the video to be segmented are divided into multiple scenes.

[0144] It can be understood that, in any scene, the scene segmentation result of each adjacent segment is used to indicate that the adjacent video segments are located in the same scene.

[0145] The technical solution provided by the above embodiment has at least the following beneficial effects: based on the multiple self-attention layers and the fully connected layer in the self-attention model, the correlation between adjacent video clips can be determined based on the correlation between the features of the clips of adjacent video clips. Subsequently, it can be determined whether two video clips to be processed are located in the same scene based on the correlation between the two video clips to be processed. In this way, an implementation method is provided that can divide all video clips into multiple scenes. At the same time, since the multiple self-attention layers in the self-attention model are respectively used to determine the correlation and dependency between the input features according to the input features, the correlation between the video clips finally calculated is more accurate, thereby ensuring that the scene division result is more accurate.

[0146] In addition, the present disclosure also provides a video segmentation device, which is used to execute the video segmentation method provided in the embodiment of the present disclosure to solve the problem of inaccurate video segmentation based on scenes in the prior art. Figure 7 As shown, the video segmentation device 40 includes an acquisition unit 401 , a determination unit 402 and a processing unit 403 .

[0147] The acquisition unit 401 is used to acquire the video to be segmented.

[0148] The determining unit 402 is configured to determine a plurality of video segments included in the video to be segmented.

[0149] The determination unit 402 is further configured to determine a segment feature of each video segment.

[0150] The processing unit 403 is configured to divide the plurality of video segments into a plurality of scenes based on the segment features of each video segment and the pre-trained self-attention model. The self-attention model is configured to determine the correlation between two adjacent video segments based on the segment features of each video segment in the two adjacent video segments.

[0151] The processing unit 403 is further configured to segment the video to be segmented based on the multiple scenes obtained by segmentation.

[0152] Optional, such as Figure 7 As shown, the determination unit 402 provided in the embodiment of the present disclosure is specifically used for:

[0153] Image features of each video segment and text features of each video segment are determined.

[0154] The segment features of each video segment are determined according to the image features and text features of each video segment.

[0155] Optional, such as Figure 7 As shown, the determination unit 402 provided in the embodiment of the present disclosure is specifically used for:

[0156] A key image frame is determined from a plurality of image frames included in each video segment, and the key image frame is located at a preset position in the plurality of image frames.

[0157] Based on the key image frame and the pre-trained first neural network, the image features of the key image frame are determined, and the image features of the key image frame are determined as the image features of each video clip. The first neural network is used to perform convolution processing on the input image frame to obtain the corresponding image features.

[0158] Determine text information of multiple image frames, and determine text features of each video segment based on the text information and a pre-trained second neural network. The second neural network is used to perform convolution processing on the input text information to obtain corresponding text features.

[0159] Optional, such as Figure 7 As shown, the determination unit 402 provided in the embodiment of the present disclosure is specifically used for:

[0160] The key image frame is cropped into a plurality of sub-image frames, and image features of the plurality of sub-image frames are generated according to the plurality of sub-image frames and the first neural network.

[0161] The image features of multiple sub-image frames are averaged and pooled to obtain the image features of the key image frame.

[0162] Optional, such as Figure 7 As shown, the processing unit 403 provided in the embodiment of the present disclosure is specifically used for:

[0163] The image features of each video clip and the text features of each video clip are input into the first self-attention layer of the self-attention model, and based on the feature weights determined by the first self-attention layer, the image features of each video clip and the text features of each video clip are weighted to obtain the first fusion features of each video clip.

[0164] For adjacent video clips among multiple video clips, the first fusion features of the adjacent video clips are input into the second self-attention layer of the self-attention model, and based on the feature weights determined by the second self-attention layer, the first fusion features of the adjacent video clips are weighted to obtain the second fusion features corresponding to the adjacent video clips.

[0165] The second fused features corresponding to adjacent video clips are input into the fully connected layer of the self-attention model to determine the correlation between adjacent video clips.

[0166] According to the correlation between adjacent video segments, a scene segmentation result for the adjacent video segments is determined. The scene segmentation result indicates whether the adjacent video segments are located in the same scene.

[0167] The multiple video segments are divided into multiple scenes according to the scene segmentation result of each adjacent video segment in the multiple video segments.

[0168] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0169] Figure 8 Schematic diagram of the structure of an electronic device provided by the present disclosure. Figure 8 The electronic device 50 may include at least one processor 501 and a memory 503 for storing processor executable instructions. The processor 501 is configured to execute instructions in the memory 503 to implement the video segmentation method in the above embodiment.

[0170] In addition, the electronic device 50 may further include a communication bus 502 and at least one communication interface 504 .

[0171] The processor 501 may be a central processing unit (CPU), a microprocessing unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the disclosed solution.

[0172] Communication bus 502 may include a pathway for transmitting information between the above-mentioned components.

[0173] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0174] The memory 503 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory may be independent and connected to the processing unit via a bus. The memory may also be integrated with the processing unit.

[0175] The memory 503 is used to store instructions for executing the solution of the present disclosure, and the execution is controlled by the processor 501. The processor 501 is used to execute the instructions stored in the memory 503, so as to realize the functions in the method of the present disclosure.

[0176] In a specific implementation, as an embodiment, the processor 501 may include one or more CPUs, such as Figure 8 CPU0 and CPU1 in.

[0177] In a specific implementation, as an embodiment, the electronic device 50 may include multiple processors, such as Figure 8 501 and processor 507 in the embodiment of the present invention. Each of these processors may be a single-CPU processor or a multi-CPU processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0178] In a specific implementation, as an embodiment, the electronic device 50 may further include an output device 505 and an input device 506. The output device 505 communicates with the processor 501 and may display information in a variety of ways. For example, the output device 505 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 506 communicates with the processor 501 and may receive user input in a variety of ways. For example, the input device 506 may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0179] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the electronic device 50, and may include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0180] In addition, the present disclosure also provides a computer-readable storage medium, including instructions, which, when executed by a processor, enable the processor to execute the video segmentation method provided in the above embodiment.

[0181] In addition, the present disclosure also provides a computer program product, including instructions, and when the instructions are executed by a processor, the processor executes the video segmentation method provided in the above embodiment.

[0182] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A video segmentation method, characterized in that: include: Acquire a video to be segmented, and determine a plurality of video segments included in the video to be segmented; Determine segment features of each video segment; the segment features include image features and text features; Inputting the image features of each video clip and the text features of each video clip into a first self-attention layer of the self-attention model, and weighting the image features of each video clip and the text features of each video clip based on the feature weights determined by the first self-attention layer to obtain a first fusion feature of each video clip; For adjacent video segments among the multiple video segments, input the first fused features of the adjacent video segments into a second self-attention layer of the self-attention model, and weight the first fused features of the adjacent video segments based on feature weights determined by the second self-attention layer to obtain second fused features corresponding to the adjacent video segments; Inputting the second fused features corresponding to the adjacent video clips into a fully connected layer of the self-attention model to determine the correlation between the adjacent video clips; Determining scene segmentation results for the adjacent video segments according to the correlation between the adjacent video segments; The scene segmentation result indicates whether the adjacent video clips are located in the same scene; Dividing the plurality of video clips into a plurality of scenes according to a scene segmentation result of each adjacent video clip in the plurality of video clips; The video to be segmented is segmented based on the multiple scenes obtained by segmentation.

2. The video segmentation method according to claim 1, characterized in that: The determining of the segment features of each video segment includes: Determining image features of each video segment and text features of each video segment; The segment features of each video segment are determined according to the image features and text features of each video segment.

3. The video segmentation method according to claim 2, characterized in that: The determining of the image features of each video segment and the text features of each video segment includes: Determining a key image frame from a plurality of image frames included in each video clip; the key image frame is located at a preset position in the plurality of image frames; Based on the key image frame and a pre-trained first neural network, determine the image features of the key image frame, and determine the image features of the key image frame as the image features of each video clip; the first neural network is used to perform convolution processing on the input image frame to obtain corresponding image features; Determine the text information of the multiple image frames, and determine the text features of each video clip based on the text information and a pre-trained second neural network; the second neural network is used to perform convolution processing on the input text information to obtain corresponding text features.

4. The video segmentation method according to claim 3, characterized in that: The determining of the image features of the key image frame based on the key image frame and the pre-trained first neural network includes: Cropping the key image frame into a plurality of sub-image frames, and generating image features of the plurality of sub-image frames according to the plurality of sub-image frames and the first neural network; The image features of the multiple sub-image frames are average-pooled to obtain the image features of the key image frame.

5. A video segmentation device, characterized in that: It includes an acquisition unit, a determination unit and a processing unit; The acquisition unit is used to acquire the video to be segmented; The determining unit is used to determine a plurality of video segments included in the video to be segmented; The determining unit is further used to determine the segment features of each video segment; the segment features include image features and text features; The processing unit is used to input the image features of each video clip and the text features of each video clip into a first self-attention layer of the self-attention model, and weight the image features of each video clip and the text features of each video clip based on the feature weights determined by the first self-attention layer to obtain a first fusion feature of each video clip; The processing unit is further configured to input, for adjacent video segments among the plurality of video segments, the first fused features of the adjacent video segments into a second self-attention layer of the self-attention model, and weight the first fused features of the adjacent video segments based on feature weights determined by the second self-attention layer to obtain second fused features corresponding to the adjacent video segments; The processing unit is further configured to input the second fused features corresponding to the adjacent video segments into a fully connected layer of the self-attention model to determine the correlation between the adjacent video segments; The processing unit is further configured to determine a scene segmentation result for the adjacent video segments according to the correlation between the adjacent video segments; the scene segmentation result indicates whether the adjacent video segments are located in the same scene; The processing unit is further configured to divide the plurality of video segments into a plurality of scenes according to a scene segmentation result of each adjacent video segment in the plurality of video segments; The processing unit is further used to segment the video to be segmented based on the multiple scenes obtained by segmentation.

6. The video segmentation device according to claim 5, characterized in that: The determining unit is specifically configured to: Determining image features of each video segment and text features of each video segment; The segment features of each video segment are determined according to the image features and text features of each video segment.

7. The video segmentation device according to claim 6, characterized in that: The determining unit is specifically configured to: Determining a key image frame from a plurality of image frames included in each video clip; the key image frame is located at a preset position in the plurality of image frames; Based on the key image frame and a pre-trained first neural network, determine the image features of the key image frame, and determine the image features of the key image frame as the image features of each video clip; the first neural network is used to perform convolution processing on the input image frame to obtain corresponding image features; Determine the text information of the multiple image frames, and determine the text features of each video clip based on the text information and a pre-trained second neural network; the second neural network is used to perform convolution processing on the input text information to obtain corresponding text features.

8. The video segmentation device according to claim 7, characterized in that: The determining unit is specifically configured to: Cropping the key image frame into a plurality of sub-image frames, and generating image features of the plurality of sub-image frames according to the plurality of sub-image frames and the first neural network; The image features of the multiple sub-image frames are average-pooled to obtain the image features of the key image frame.

9. An electronic device, characterized in that: include: A processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute instructions so that the electronic device implements the video segmentation method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the video segmentation method according to any one of claims 1 to 4.

11. A computer program product comprising instructions, characterized in that The computer program product includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the video segmentation method according to any one of claims 1 to 4.

Images