Video scene segmentation method, device, computer equipment, and storage medium

Abstract

The present application relates to a method, apparatus, computer device, storage medium, and computer program product for segmenting a video scene. The computer device may include a smartphone, a computer, or an intelligent vehicle-mounted device; the method includes: extracting features from a video shot sequence using a dual-path model to obtain shot features; determining positive sample features from the shot features; obtaining negative sample features, and optimizing a first encoder and a second encoder based on the loss value between the negative sample features and the positive sample features; wherein the optimized first encoder serves as a shot feature extraction model; extracting a third shot feature from a target video shot sequence using the shot feature extraction model, and training a scene segmentation model based on the third shot feature; and performing video scene segmentation on the video to be segmented based on the trained scene segmentation model. This method can effectively improve the accuracy of video scene segmentation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for segmenting a video scene. Background Art

[0002] With the development of computer technology and Internet technology, video scene segmentation is an important technology in video understanding. It mainly uses a long video as a segmentation clue according to the different scene contents, and divides the long video into several independent sub-videos based on the shot dimension. For this technology, extracting good shot features and selecting a suitable shot segmentation model are crucial.

[0003] However, the current video scene segmentation method mainly adopts the method of aggregating a number of adjacent shot features on both sides of a shot boundary to obtain the features of the shot boundary, and then further classifying the features of the shot boundary to decide whether the boundary is a scene boundary. The problem brought about by this is that the number of shot features aggregated at each shot boundary is uncertain, and the number of shots in each scene is also not fixed. Therefore, a strong inductive bias will be introduced, which can easily lead to poor generalization ability of the video scene segmentation model and poor accuracy of video scene segmentation. Summary of the Invention

[0004] Based on this, it is necessary to provide a video scene segmentation method, apparatus, computer equipment, computer-readable storage medium and computer program product that can effectively improve the accuracy of video scene segmentation in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for segmenting a video scene. The method comprises: extracting features from a video shot sequence using a first encoder and a second encoder of a dual-path model, respectively, to obtain first shot features and second shot features; determining a first sample feature in the first shot features, and determining a second sample feature in the second shot features based on the encoding of the first sample feature; using the first sample feature and the second sample feature as positive sample features; obtaining negative sample features, and optimizing the first encoder and the second encoder based on the loss value between the negative sample feature and the positive sample feature; wherein the optimized first encoder serves as a shot feature extraction model; extracting a third shot feature from a target video shot sequence using the shot feature extraction model, and training a scene segmentation model based on the third shot feature; and performing video scene segmentation on the video to be segmented based on the trained scene segmentation model.

[0006] In the second aspect, the present application also provides a video scene segmentation device. The device includes: an extraction module, which is used to extract features from a video shot sequence through a first encoder and a second encoder of a dual-path model, respectively, to obtain a first shot feature and a second shot feature; a determination module, which is used to determine a first sample feature in the first shot feature, and determine a second sample feature in the second shot feature based on the encoding of the first sample feature; the first sample feature and the second sample feature are used as positive sample features; an optimization module, which is used to obtain a negative sample feature, and optimize the first encoder and the second encoder based on the loss value between the negative sample feature and the positive sample feature; wherein the optimized first encoder is used as a shot feature extraction model; a training module, which is used to extract a third shot feature from a target video shot sequence through the shot feature extraction model, and train a scene segmentation model based on the third shot feature; a segmentation module, which is used to perform video scene segmentation on the video to be segmented based on the trained scene segmentation model.

[0007] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: extracting features from a video shot sequence using a first encoder and a second encoder of a dual-path model, respectively, to obtain first shot features and second shot features; determining a first sample feature in the first shot features, and determining a second sample feature in the second shot features based on the encoding of the first sample feature; using the first sample feature and the second sample feature as positive sample features; obtaining a negative sample feature, and optimizing the first encoder and the second encoder based on the loss value between the negative sample feature and the positive sample feature; wherein the optimized first encoder serves as a shot feature extraction model; extracting a third shot feature from a target video shot sequence using the shot feature extraction model, and training a scene segmentation model based on the third shot feature; and performing video scene segmentation on the video to be segmented based on the trained scene segmentation model.

[0008] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps: extracting features from a video shot sequence using a first encoder and a second encoder of a dual-path model, respectively, to obtain first shot features and second shot features; determining a first sample feature in the first shot features, and determining a second sample feature in the second shot features based on the encoding of the first sample feature; using the first sample feature and the second sample feature as positive sample features; obtaining a negative sample feature, and optimizing the first encoder and the second encoder based on the loss value between the negative sample feature and the positive sample feature; wherein the optimized first encoder serves as a shot feature extraction model; extracting a third shot feature from a target video shot sequence using the shot feature extraction model, and training a scene segmentation model based on the third shot feature; and performing video scene segmentation on the video to be segmented based on the trained scene segmentation model.

[0009] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps: extracting features from a video shot sequence using a first encoder and a second encoder of a dual-path model, respectively, to obtain first shot features and second shot features; determining a first sample feature in the first shot features, and determining a second sample feature in the second shot features based on the encoding of the first sample feature; using the first sample feature and the second sample feature as positive sample features; obtaining negative sample features, and optimizing the first encoder and the second encoder based on the loss value between the negative sample feature and the positive sample feature; wherein the optimized first encoder serves as a shot feature extraction model; extracting a third shot feature from a target video shot sequence using the shot feature extraction model, and training a scene segmentation model based on the third shot feature; and performing video scene segmentation on the video to be segmented based on the trained scene segmentation model.

[0010] The above-mentioned video scene segmentation method, device, computer equipment, storage medium and computer program product respectively extract features of a video shot sequence through the first encoder and the second encoder of a dual-path model to obtain first shot features and second shot features; determine a first sample feature in the first shot features, and determine a second sample feature in the second shot features based on the encoding of the first sample feature; the first sample feature and the second sample feature are used as positive sample features; obtain negative sample features, and optimize the first encoder and the second encoder based on the loss value between the negative sample features and the positive sample features; wherein the optimized first encoder is used as a shot feature extraction model; extract a third shot feature from the target video shot sequence through the shot feature extraction model, and train a scene segmentation model based on the third shot feature; and perform video scene segmentation on the video to be segmented based on the trained scene segmentation model. Since the first encoder and the second encoder of the dual-path model respectively extract features of the video shot sequence, the best positive sample features can be selected from the extracted first shot features and the second shot features, and then the shot feature extraction model is optimized using the selected positive sample features and negative sample features, so that the optimized shot feature extraction model has more accurate recognition ability and can extract good shot features. The scene segmentation model is then trained based on the shot features extracted by the lens feature extraction model, so that the trained scene segmentation model can accurately and efficiently perform video scene segmentation on the video to be segmented, thereby effectively improving the accuracy of video scene segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 FIG. 1 is an application environment diagram of a method for segmenting a video scene in one embodiment;

[0012] Figure 2 1 is a flow chart of a method for segmenting a video scene in one embodiment;

[0013] Figure 3 FIG1 is a flow chart showing the steps of shuffling video shots in a continuous video shot sequence to obtain a shuffled video shot sequence in one embodiment;

[0014] Figure 4 Schematic diagram of a shot-independent shot sequence scrambling method according to one embodiment;

[0015] Figure 5 Schematic diagram showing a comparison between a scene consistency selection method and a nearest neighbor selection method in a video scene segmentation task according to an embodiment;

[0016] Figure 6 1 is a schematic diagram of the overall framework of a consistent representation learning method for video scene segmentation in one embodiment;

[0017] Figure 7Schematic diagram showing a comparison between a scene boundary model and a scene free model in one embodiment;

[0018] Figure 8 1 is a flow chart of a training strategy in a consistent representation learning method framework for video scene segmentation in one embodiment;

[0019] Figure 9 is a structural block diagram of a video scene segmentation device in one embodiment;

[0020] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0022] The video scene segmentation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other servers. The server 104 can obtain the video shot sequence uploaded by the terminal 102, or the server 104 can obtain the video shot sequence from the database. The server 104 extracts features from the video shot sequence through the first encoder and the second encoder of the dual-path model, respectively, to obtain first shot features and second shot features; the server 104 determines the first sample feature in the first shot feature, and determines the second sample feature in the second shot feature based on the encoding of the first sample feature; the first sample feature and the second sample feature are used as positive sample features; the server 104 obtains the negative sample feature, and optimizes the first encoder and the second encoder based on the loss value between the negative sample feature and the positive sample feature; wherein the optimized first encoder is used as a shot feature extraction model; the server 104 extracts the third shot feature from the target video shot sequence through the shot feature extraction model, and trains the scene segmentation model based on the third shot feature; the server 104 performs video scene segmentation on the video to be segmented based on the trained scene segmentation model.

[0023] The terminal 102 may be, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.

[0024] The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers. It is understood that the server 104 provided in the embodiment of the present application can also be a service node in the blockchain system. The service nodes in the blockchain system form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP).

[0025] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0026] Cloud technology is a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Different levels of data will be processed separately, and data from various industries will require a strong system backend, which can only be achieved through cloud computing.

[0027] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0028] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0029] Deep learning, a branch of machine learning, is an algorithm that attempts to achieve high-level abstraction of data using multiple processing layers containing complex structures or consisting of multiple nonlinear transformations. Deep learning is an algorithm in machine learning that learns to represent data. To date, several deep learning frameworks, such as convolutional neural networks, deep belief networks, and recurrent neural networks, have been applied in fields such as computer vision, speech recognition, natural language processing, audio recognition, and bioinformatics, achieving excellent results.

[0030] In one embodiment, Figure 2 As shown, a video scene segmentation method is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the server in the example:

[0031] Step 202 : extract features from the video shot sequence using the first encoder and the second encoder of the dual-path model to obtain first shot features and second shot features.

[0032] Among them, the two-way model refers to a two-way training branch model composed of two encoders, usually called a twin network. The two-way model in this application is mainly used to measure the similarity of two input video shot sequences.

[0033] The first encoder refers to an encoder in the two-way model. For example, the first encoder may be a query encoder in the two-way model.

[0034] The second encoder refers to another encoder in the two-way model. For example, the second encoder may be a key-value encoder in the two-way model.

[0035] A video shot sequence is a sequence consisting of at least two video shots, where a shot contains continuous frames captured by a camera without interruption. A scene consists of continuous shots, describing the same short story. A continuous shot sequence of any length can generally be considered a video short. The video shot sequence in this application contains at least two video shots, each of which can contain a different number of continuous video frames. It is understood that when training a two-way model in this application, the video shot sequence input into the two-way model is an unlabeled video shot sequence.

[0036] The first shot feature is a shot feature extracted by the first encoder from the video shot sequence. For example, the first encoder may be a query encoder in a two-way model, and the first shot feature is the shot feature extracted by the query encoder.

[0037] The second shot feature is a shot feature extracted by the second encoder from the video shot sequence. For example, the second encoder may be a key-value encoder in a two-way model, and the second shot feature is the shot feature extracted by the key-value encoder.

[0038] Specifically, a camera is installed in the terminal device, and the terminal can use the camera to collect video shot sequences in various scenarios and upload them to the server. Alternatively, if the server is associated with other devices, the server can obtain video shot sequences collected by other devices. For example, if the server is associated with a vehicle, the server can obtain video shot sequences collected by the target vehicle on the road. The method for the server to obtain the video shot sequence is not specifically limited here, and other acquisition methods can also be used. The video shot sequence can be a video shot sequence collected in real time, or a video shot sequence that has been collected and is to be processed. For example, the server can perform pre-processing such as cropping on the original video shot sequence, and then use the pre-processed video shots as the input video shot sequence to input into the dual-path model.

[0039] After the server obtains the video shot sequence, the server may respectively perform feature extraction on the video shot sequence through the first encoder and the second encoder of the dual-path model to obtain first shot features and second shot features.

[0040] For example, suppose that the video shot sequence A obtained by the server is: shot 1 → shot 2 → shot 3, that is, the video shot sequence contains 3 video shots, namely shot 1, shot 2 and shot 3, and the number of video frames contained in each shot is different. For example, shot 1 contains 10 continuous frames, shot 2 contains 20 continuous frames, and shot 3 contains 25 continuous frames. The server uses the above-obtained video shot sequence as the input sequence, and extracts features of the video shot sequence A through the first encoder and the second encoder of the dual-path model respectively to obtain the first shot feature and the second shot feature.

[0041] Step 204 : determining a first sample feature in the first shot feature, and determining a second sample feature in the second shot feature based on the encoding of the first sample feature; the first sample feature and the second sample feature are used as positive sample features.

[0042] The first sample feature refers to a target shot feature selected from the first shot feature as the first sample feature. For example, the extracted first shot feature includes feature A and feature B. Based on a preset sample selection strategy, feature A is selected as the first sample feature.

[0043] The encoding of the first sample feature refers to the identification information in the training batch to which the first sample feature belongs. For example, if the identification information in the training batch to which feature A in the first sample feature belongs is c3, it is recorded as i c3 .

[0044] The second sample feature refers to a target shot feature selected from the second shot feature as the second sample feature. For example, the extracted second shot feature includes feature A and feature B. Based on the preset sample selection strategy, feature B is selected as the second sample feature.

[0045] A positive sample feature refers to a sample feature belonging to a certain category. The positive sample feature is used as an input positive sample when optimizing the first encoder and the second encoder of the dual-channel model. The positive sample feature in this application represents that the shot boundary is a scene boundary.

[0046] Specifically, the server extracts features from the video shot sequence through the first encoder and the second encoder of the dual-path model respectively. After obtaining the first shot feature and the second shot feature, the server can determine the first sample feature in the first shot feature based on a preset positive sample selection strategy, and determine the second sample feature in the second shot feature based on the encoding of the first sample feature. Finally, the server uses the first sample feature and the second sample feature as a positive sample feature pair.

[0047] For example, the server can first cluster the first shot features extracted by the first encoder based on the scene consistency positive sample selection strategy to obtain several clusters and the central sample index of each cluster. For the i-th sample in each cluster, the central sample of its cluster is selected as its corresponding positive sample, and its index value in the batch is recorded as the mapping function MAP(i). The server can determine another positive sample in the second encoder based on the mapping function MAP(i), thereby forming a positive sample pair.

[0048] For example, suppose the video shot sequence A obtained by the server is: shot 1 → shot 2 → shot 3, that is, the video shot sequence contains 3 video shots, namely shot 1, shot 2 and shot 3. The server uses the above-obtained video shot sequence as the input sequence, and uses the first encoder and the second encoder of the dual-path model to extract features of the video shot sequence A respectively. The extracted first shot features are a1, b1, c1 and d1; the extracted second shot features are a2, b2, c2 and d2.

[0049] Furthermore, the server can cluster the first shot features a1, b1, c1, and d1 extracted by the first encoder based on the positive sample selection strategy of scene consistency, and obtain two clusters. Cluster 1 contains: a1 and c1; Cluster 2 contains: b1 and d1; and the central sample subscript of each cluster. The central sample subscript of cluster 1 is: i a1 , the central sample of cluster 2 is subscripted as: i b1 For each i-th sample in a cluster, the center sample of the cluster is selected as its corresponding positive sample. For example, for the two samples a1 and c1 contained in cluster 1, the center sample subscript i of the cluster is selected. a1 The corresponding center sample a1 is taken as a positive sample, and the index value of the center sample a1 in its training batch is recorded as the mapping function MAP(i). The server can determine another positive sample a2 in the second lens features a2, b2, c2 and d2 based on the mapping function MAP(i), thereby forming a positive sample pair a1 and a2.

[0050] Step 206: Obtain negative sample features, and optimize the first encoder and the second encoder based on the loss value between the negative sample features and the positive sample features; wherein the optimized first encoder is used as a shot feature extraction model.

[0051] Among them, negative sample features refer to sample features that do not belong to a certain category. Negative sample features are used to use negative sample features as input negative samples when optimizing the first encoder and the second encoder of the dual-channel model. The negative sample features in this application represent that the shot boundary is a non-scene boundary.

[0052] The loss value refers to the loss value between the negative sample features and the positive sample features when training the two-way model. In this application, a designed contrast loss function is used to optimize the first encoder and the second encoder.

[0053] The lens feature extraction model refers to a neural network model used to extract lens features. For example, the ResNet50 network can be used as the basic network framework of the lens feature extraction model in this application.

[0054] Specifically, after the server determines the first sample feature in the first shot feature and determines the second sample feature in the second shot feature based on the encoding of the first sample feature, the server can obtain the negative sample feature, optimize the first encoder and the second encoder in the two-way model based on the loss value between the negative sample feature and the positive sample feature, and use the optimized first encoder as the trained shot feature extraction model.

[0055] The negative sample features in this application can be cached using a fixed-length first-in, first-out queue, caching the second-shot features generated by the second encoder in the previous pass as historical samples. The historical samples in this queue are then used as negative sample features in the next optimization process. Furthermore, when the server optimizes the first and second encoders in the two-pass model for the first time, it can directly retrieve the negative sample features randomly generated during the initialization of this queue as the negative sample features used for the first optimization of the two-pass model.

[0056] It is understood that the method of obtaining negative samples in this application includes but is not limited to the above-mentioned method, and other customized methods can also be used to obtain corresponding negative sample features.

[0057] For example, the first training of a two-way model is used as an example. When the two-way model is trained for the first time, the server extracts features from the video shot sequence through the first encoder and the second encoder of the two-way model respectively, obtains the first shot features and the second shot features, and after determining the positive sample features from the first shot features and the second shot features, the server can initialize the negative sample queue and generate random negative samples, that is, generate a vector of random numbers that obey a Gaussian distribution as negative sample features. The server can optimize the first encoder and the second encoder in the two-way model based on the loss value between the negative sample features and the positive sample features obtained above, and use the optimized first encoder as the trained shot feature extraction model.

[0058] Step 208: extracting a third shot feature from the target video shot sequence using a shot feature extraction model, and training a scene segmentation model based on the third shot feature.

[0059] Among them, the target video shot sequence refers to the input sequence used to train the scene segmentation model. For example, the target video shot sequence in this application can be a labeled video shot sequence, which is different from the training of the two-way model in this application, in which the video shot sequence input into the two-way model is an unlabeled video shot sequence.

[0060] The third shot feature refers to the shot feature extracted from the target video shot sequence by the shot feature extraction model, that is, the third shot feature, which is different from the first shot feature and the second shot feature extracted from the unlabeled video shot sequence in the input dual-path model.

[0061] The scene segmentation model refers to a model used to divide a video into several independent sub-videos in the time domain according to the different scenes in the video. For example, in this application, Bi-LSTM and three-layer fully connected layers can be used as the basic network framework of the scene segmentation model.

[0062] Specifically, the server optimizes the first encoder and the second encoder based on the loss value between the negative sample features and the positive sample features, and uses the optimized first encoder as the shot feature extraction model. The server can then extract the third shot feature from the target video shot sequence through the shot feature extraction model, and train the scene segmentation model based on the third shot feature. That is, the server can extract the third shot feature from the target video shot sequence through the optimized first encoder, and train the scene segmentation model based on the third shot feature. The corresponding output sequence represents the scene boundary label prediction sequence, and uses the cross-entropy loss function as the optimization target. That is, the server can use the cross-entropy loss function to optimize the scene segmentation model, and use the optimized scene segmentation model as the final trained scene segmentation model.

[0063] In the training phase of the video scene segmentation model, the purpose is to perform supervised training on the video scene segmentation model. In this phase, the server uses the first encoder optimized in the previous phase as the shot feature extraction model, that is, the parameters of the shot feature extraction model used are fixed. For example, the server first inputs the labeled video shot sequence B into the first encoder optimized in the previous phase, and extracts the corresponding embedded feature C from the labeled video shot sequence B through the optimized first encoder. The server then inputs the embedded feature C into the designed scene segmentation model, and the corresponding output sequence represents the scene boundary label prediction sequence. At the same time, during the training process, the server can use the cross-entropy loss function as the optimization target and use the gradient back-propagation strategy to update the network parameters of the scene segmentation model.

[0064] Step 210 : performing video scene segmentation on the video to be segmented based on the trained scene segmentation model.

[0065] The video to be segmented refers to the video selected by the user for scene segmentation. The video to be segmented can contain multiple video shots. For example, the video to be segmented in this application contains at least two video shots, each of which contains multiple consecutive video frames. For example, if the video to be segmented is: shot 1 → shot 2 → shot 3, the video to be segmented contains three video shots.

[0066] Specifically, the server extracts the third shot feature from the target video shot sequence through the shot feature extraction model, and after training the scene segmentation model based on the third shot feature, the server can perform video scene segmentation on the video to be segmented based on the trained scene segmentation model. That is, the trained scene segmentation model in this application can be applied to long video scene segmentation tasks, which is usually defined as video stripping capability on the product side, that is, given a long video, the scene segmentation model trained in this application can automatically divide the video into several independent sub-videos in the time domain according to the different scenes in the video. In addition, the trained scene segmentation model in this application can also be applied to video shot retrieval, that is, for a given shot, shots of similar scenes are automatically retrieved and sorted according to similarity.

[0067] For example, taking a target video shot sequence with an input length of B*Shot-Len*N as an example, where B in B*Shot-Len*N is the training batch size, Shot-Len is the number of shots processed in a single batch, and N is the dimension of the shot feature, the model training method described above in this application is used to classify each input shot as a scene boundary, i.e., to determine whether each shot in the input target video shot sequence is a scene boundary. If a shot in the target video shot sequence is determined to be a scene boundary, the sequence value corresponding to the output shot is 1. If a shot in the target video shot sequence is determined to be a non-scene boundary, the sequence value corresponding to the output shot is 0. That is, when the sequence value of the output sequence is 0, it indicates a non-scene boundary; when the sequence value of the output sequence is 1, it indicates a scene boundary. That is, after the Bi-LSTM model sequence modeling in this application, the scene segmentation model outputs a sequence of B*Shot-Len*2, and each sequence value in the output sequence represents whether the boundary of the shot is a scene boundary.

[0068] For example, assuming that the video A to be segmented is: shot 1→shot 2→shot 3, that is, the video to be segmented contains 3 video shots, then the server inputs the video A to be segmented into the trained scene segmentation model, and the output sequence is 110, which means that shot 1 in the video A to be segmented is a scene boundary, shot 2 is also a scene boundary, and shot 3 is a non-scene boundary. After video scene segmentation of the video A to be segmented, shot 1 is one video scene, and shot 2→shot 3 are another video scene.

[0069] In the above-mentioned video scene segmentation method, the first encoder and the second encoder of the dual-path model are used to extract features of the video shot sequence respectively to obtain the first shot feature and the second shot feature, determine the first sample feature in the first shot feature, and determine the second sample feature in the second shot feature based on the encoding of the first sample feature, and the first sample feature and the second sample feature are used as positive sample features; obtain the negative sample feature, and optimize the first encoder and the second encoder based on the loss value between the negative sample feature and the positive sample feature; wherein the optimized first encoder is used as a shot feature extraction model; extract the third shot feature from the target video shot sequence through the shot feature extraction model, and train the scene segmentation model based on the third shot feature, and perform video scene segmentation on the video to be segmented based on the trained scene segmentation model. Since the first encoder and the second encoder of the dual-path model respectively extract features of the video shot sequence, the best positive sample features can be selected from the extracted first shot features and the second shot features, and then the shot feature extraction model is optimized using the selected positive sample features and negative sample features, so that the optimized shot feature extraction model has more accurate recognition ability and can extract good shot features. The scene segmentation model is then trained based on the shot features extracted by the lens feature extraction model, so that the trained scene segmentation model can accurately and efficiently perform video scene segmentation on the video to be segmented, thereby effectively improving the accuracy of video scene segmentation.

[0070] In one embodiment, before extracting features from the video shot sequence using the first encoder and the second encoder of the two-way model, the method further includes:

[0071] Acquire temporally continuous video shots to obtain a continuous video shot sequence;

[0072] Performing random processing on the video shots in the continuous video shot sequence to obtain a random video shot sequence;

[0073] Data enhancement processing is performed on each video shot in the disordered video shot sequence to obtain a video shot sequence for feature extraction.

[0074] A video shot is a sequence of continuous frames captured by a camera without interruption. Each video shot can contain a varying number of consecutive video frames. A scene consists of consecutive shots, describing the same short story. Generally, a sequence of shots of any length can be considered a clip.

[0075] A continuous video shot sequence refers to a video shot sequence consisting of video shots that are continuous in time. For example, a continuous video shot sequence is: shot 1 → shot 2 → shot 3, which means that shot 1, shot 2, and shot 3 are continuous in time.

[0076] Shuffle processing refers to disrupting the order of each video frame in the original data to obtain the shuffled data. For example, the original data to be input is a→b→c, and the data obtained after shuffle processing is b→a→c.

[0077] Data augmentation processing is used to increase the diversity of training samples. Data augmentation is mainly to reduce the overfitting phenomenon of the network. By transforming the training images, a network with stronger generalization ability can be obtained, which can better adapt to the application scenario. For example, the data augmentation processing method used in this application can be random data augmentation processing, including random cropping, Gaussian blur, random scaling, random color transformation, random grayscale transformation, and random horizontal inversion operations.

[0078] Specifically, before the server extracts features from the video shot sequence using the first encoder and the second encoder of the dual-path model, the server can obtain temporally continuous video shots to obtain a continuous video shot sequence, and use the continuous video shot sequence as the video shot sequence to be input. Furthermore, the server can perform shuffling and data enhancement processing on the input video shot sequence.

[0079] In the learning and training stage of shot features, data enhancement is crucial. The data enhancement methods used in this application can be divided into two types. One is to regularly shuffle the input shot sequence, which is called scene-independent shot sequence shuffling. The other is to perform asymmetric image enhancement processing on a single shot in the input shot sequence. In the scene-independent shot sequence shuffling method, since the scene boundary labels cannot be obtained at the input shot sequence stage, in order to improve the richness of shots and scenes in each training batch, it is necessary to shuffle the entire input shot sequence and perform asymmetric image enhancement processing on a single shot in the input shot sequence. Therefore, by regularly shuffling the entire input shot sequence and then performing asymmetric image enhancement processing on a single shot in the shuffled shot sequence, the richness of shots and scenes in each training batch can be effectively improved, the diversity of training samples can be increased, and the overfitting phenomenon of the network can be reduced. By transforming the training images, a network with stronger generalization ability can be obtained, which is better adapted to the application scenario of long video scene segmentation tasks.

[0080] In one embodiment, Figure 3 As shown, the steps of shuffling the video shots in the continuous video shot sequence to obtain a shuffled video shot sequence include:

[0081] Step 302: Divide each video shot in the continuous video shot sequence into segments to obtain video segments;

[0082] Step 304 : Shuffle the video clips of each video shot in the continuous video shot sequence to obtain a shuffled video shot sequence.

[0083] Among them, segmentation refers to dividing the video frames in each video shot in a continuous video shot sequence according to a preset strategy to obtain the divided video segments. For example, according to the preset strategy, the video frames in each video shot in a continuous video shot sequence are divided as a video segment with every 5 video frames, so that multiple video segments with every 5 frames are obtained.

[0084] Specifically, before the server extracts features from the video shot sequence using the first and second encoders of the dual-path model, it can obtain temporally continuous video shots to obtain a continuous video shot sequence, and use this continuous video shot sequence as the input video shot sequence. Furthermore, the server can shuffle the input video shot sequence. Specifically, the server can segment each video shot in the continuous video shot sequence according to a preset segment segmentation strategy to obtain multiple corresponding video segments, and then shuffle each video segment to obtain a shuffled video shot sequence.

[0085] For example, suppose the preset segment division strategy is: use continuous shots of fixed length ρ as independent units and shuffle the entire shot sequence. Figure 4 , which is a schematic diagram of a shot-independent shot sequence scrambling method. Figure 4 The original shot sequence in is: video A → video B → video C, that is, the original shot sequence contains three shot scenes. Assuming ρ = 2, using a continuous shot of a fixed length of ρ as an independent unit, after shuffling the entire original shot sequence, we can get the following: Figure 4 The server can send the out-of-order video shot sequence shown in Figure 4 The out-of-order video shot sequence shown in is used as the video shot sequence for training in the input dual-path model, as shown in Figure 4 As shown, during each training session, the server can extract four frames from the shuffled video shot sequence as the video shot sequence to be input into the dual-channel model for this training batch. Therefore, by using fixed-length continuous shots as independent units and shuffling the entire shot sequence, the richness of shots and scenes in each training batch can be effectively improved, increasing the diversity of training samples and reducing network overfitting. By transforming the training images, a network with stronger generalization capabilities can be obtained, which is better suited for the application scenario of long video scene segmentation tasks.

[0086] In one embodiment, the step of shuffling video shots in a continuous video shot sequence to obtain a shuffled video shot sequence includes:

[0087] Obtain at least one video shot to be inserted;

[0088] At least one video shot to be inserted is inserted between adjacent video shots in a video shot sequence to obtain a disordered video shot sequence.

[0089] Specifically, before the server performs feature extraction on the video shot sequence using the first and second encoders of the dual-path model, the server may obtain temporally continuous video shots to obtain a continuous video shot sequence, and use this continuous video shot sequence as the video shot sequence to be input. Furthermore, the server may shuffle the input video shot sequence. Specifically, the server may obtain at least one video shot to be inserted according to a preset shuffling strategy, and insert the at least one video shot to be inserted between adjacent video shots in the video shot sequence, thereby obtaining a shuffled video shot sequence.

[0090] For example, suppose the preset scrambling strategy is to obtain two video shots with two frames to be inserted. That is, the server can obtain video shot 1 and video shot 2, and each of video shot 1 and video shot 2 contains two consecutive frames. Furthermore, the server can randomly insert the obtained video shots 1 and video shot 2 between adjacent video shots in the original video shot sequence to obtain a scrambled video shot sequence. Therefore, by scrambling the entire shot sequence, the richness of shots and scenes in each training batch can be effectively improved, the diversity of training samples can be increased, and the overfitting of the network can be reduced. By transforming the training images, a network with stronger generalization ability can be obtained, which is better adapted to the application scenario of long video scene segmentation tasks.

[0091] In one embodiment, the step of performing data enhancement processing on each video shot in the out-of-order video shot sequence to obtain a video shot sequence for feature extraction includes:

[0092] performing a first data augmentation process on each video shot in the disordered video shot sequence to obtain a video shot sequence for input into a first encoder for feature extraction; the first data augmentation process includes at least one of a cropping process, a blurring process, and a reversal operation;

[0093] A second data augmentation process is performed on each video shot in the disordered video shot sequence to obtain a video shot sequence for input into a second encoder for feature extraction; the second data augmentation process includes at least one of scaling, cropping, pixel transformation, blurring, and horizontal inversion operations.

[0094] Specifically, in contrastive learning methods, a twin network structure is often used as the basic design paradigm of the network. Since the dual-path model in this application has dual-path branches, for the same sample input, after random data enhancement, two view samples of the sample are obtained, and input into the dual-path model to generate two representations of the sample. In this application, the consistency constraint loss, that is, the contrastive learning loss, is used to gradually approach the two representations in spatial distribution, that is, to achieve the purpose of making the dual-path network model insensitive to various transformation operations of the same sample and producing relatively consistent representations.

[0095] At the same time, due to the drastic changes in data, the selection of positive samples in the early stages of learning is greatly interfered with. For example, in the early stages of the two-way model, it is easier to select the wrong positive sample pairs. For example, if the samples in a training batch are randomly changed in color, the two-way model is more likely to choose sample pairs with the same color as the positive sample pairs in the early stages, rather than focusing on the differences and similarities in semantic information to select positive sample pairs. Among them, the data enhancement methods for drastic changes include: data enhancement for color transformation, i.e., random color transformation, random grayscale transformation, and large-scale random scaling. In addition, the positive samples in this application are selected in the first encoder branch, so the data enhancement method used in the first encoder branch is simpler, and relative to the other branch, i.e., the second encoder, there is no need to perform random scaling, random color transformation, and random grayscale transformation.

[0096] Therefore, in this embodiment, before the server performs feature extraction on the video shot sequence using the first and second encoders of the dual-path model, the server can obtain temporally continuous video shots to obtain a continuous video shot sequence, and use this continuous video shot sequence as the video shot sequence to be input. The server can then shuffle the input video shot sequence to obtain a shuffled video shot sequence. Furthermore, the server can perform data augmentation processing on each video shot in the shuffled video shot sequence to obtain a video shot sequence for feature extraction. Specifically, the server can crop, blur, and invert each video shot in the shuffled video shot sequence to obtain a video shot sequence for input to the first encoder for feature extraction. Simultaneously, the server can scale, crop, pixel-transform, blur, and horizontally invert each video shot in the shuffled video shot sequence to obtain a video shot sequence for input to the second encoder for feature extraction.

[0097] In this embodiment, compared to using the same data augmentation method, which reduces the stability of model learning and the ultimate generalization, the embodiment of the present application uses different data augmentation methods for the first encoder and the second encoder in the dual-path model, so that more accurate positive samples can be automatically selected when training the dual-path model, thereby making the trained shot feature extraction model have more accurate recognition capabilities, effectively improving the generalization ability of the shot feature extraction model, that is, effectively improving the stability of model learning and the ultimate generalization.

[0098] In one embodiment, the step of determining the first sample feature in the first shot feature includes:

[0099] Performing clustering processing on the first shot feature to obtain shot feature clusters and central shot features of each shot feature cluster;

[0100] Select the central shot feature of each shot feature cluster as the first sample feature;

[0101] The method further includes: recording the code of the central shot feature of each shot feature cluster.

[0102] Clustering refers to the process of dividing a collection of physical or abstract objects into multiple classes consisting of similar objects. The clusters generated by clustering are a collection of data objects that are similar to objects in the same cluster and different from objects in other clusters. For example, the clustering process in this application can use the K-Means algorithm to perform clustering, thereby obtaining several clusters and the central sample index of each cluster. That is, the server divides a dataset, i.e., the first lens feature dataset, into different classes or clusters according to a specific criterion, such as distance, so that the similarity of lens features within the same cluster is as great as possible, while the difference of lens features not in the same cluster is as great as possible.

[0103] Lens feature clustering refers to dividing the extracted multiple lens features into different clusters according to preset specific standards. The lens features in each cluster are similar to each other, and each cluster after segmentation is a lens feature cluster.

[0104] The central lens feature refers to the central lens feature in each lens feature cluster. For example, a lens feature cluster contains 5 lens features, wherein the third lens feature in the lens feature cluster is the central lens feature.

[0105] The first sample feature refers to a target shot feature that can be used as a positive sample and is determined from multiple first shot features, that is, the first sample feature.

[0106] The encoding of the central lens feature refers to the identification information corresponding to the central lens feature, that is, the identification information of each central lens feature in the training batch. For example, the identification information of the training batch to which the central lens feature A belongs is c2, which is recorded as i c2 .

[0107] Specifically, the server extracts features from the video shot sequence using the first encoder and the second encoder of the dual-path model respectively. After obtaining the first shot features and the second shot features, the server can cluster the obtained first shot features to obtain corresponding shot feature clusters and the central shot features of each shot feature cluster; further, the server can select the central shot features of each shot feature cluster as the first sample feature, and record the encoding of the central shot features of each shot feature cluster.

[0108] For example, suppose the first shot feature extracted by the first encoder is recorded as f(·|θ Q ), the second shot feature extracted by the second encoder is f(·|θ K ), where θ Q and θ K The server first uses the K-Means algorithm to cluster the first shot features extracted by the first encoder, and obtains several clusters and the central sample subscript I of each cluster. C ={i c1 ,i c2 ,i c3 ,…,i c#class}, where #class is the number of clusters. For the i-th sample in each cluster, the center sample of the cluster is selected as its corresponding positive sample, and its index value in the training batch is recorded to facilitate the subsequent generation of the corresponding mapping function MAP(i). For example, a cluster contains 5 lens features, and the center lens feature of the cluster is the third lens feature, which is recorded as i c3 , then for each shot feature in the cluster, select the central sample i of its cluster c3 As its corresponding positive sample, and record its index value in the training batch as c3.

[0109] In this embodiment, compared with the positive sample selection method of nearest neighbor selection used in the traditional method, this embodiment clusters the first lens features and selects positive samples from each cluster after clustering, that is, uses a positive sample selection method based on scene consistency selection, so that more accurate positive samples can be automatically selected when training the two-way model, so that the trained lens feature extraction model has more accurate recognition ability, effectively improves the generalization ability of the lens feature extraction model, and solves the problems of poor generalization ability of the lens feature extraction model and poor accuracy of video scene segmentation.

[0110] In one embodiment, the step of determining the second sample feature in the second shot feature based on the encoding of the first sample feature includes:

[0111] The code is used as a variable of the index mapping function; the index mapping function is used to reflect the mapping relationship between the central lens feature of each lens feature cluster and the second sample feature;

[0112] Based on the index mapping function, a second sample feature is determined from the second lens feature.

[0113] The second sample feature refers to a target shot feature that can be used as a positive sample and is determined from multiple second shot features, that is, the second sample feature.

[0114] Specifically, after the server determines the first sample feature in the first lens feature, the server can determine the second sample feature in the second lens feature based on the encoding of the first sample feature, that is, the server can use the encoding of the central lens feature of each lens feature cluster as a variable of the index mapping function, and determine the corresponding second sample feature from the second lens feature based on the index mapping function; wherein the index mapping function is used to reflect the mapping relationship between the central lens feature of each lens feature cluster and the second sample feature. Therefore, the server can select a target lens feature as a positive sample from multiple second lens features based on the index mapping function, and the target lens feature is the second sample feature.

[0115] For example, suppose the first shot feature extracted by the first encoder is recorded as f(·|θ Q ), the second shot feature extracted by the second encoder is f(·|θ K ), where θ Q and θ K The server first uses the K-Means algorithm to cluster the first shot features extracted by the first encoder, and obtains several clusters and the central sample subscript I of each cluster. C ={i c1 ,i c2 ,i c3 ,…,i c#class}, where #class is the number of class clusters. For the i-th sample in each cluster, the center sample of the cluster is selected as its corresponding positive sample, and its index value in the training batch is recorded as the index mapping function MAP(i). Furthermore, the server can determine the actual positive sample in the shot features generated by the second encoder based on the index mapping function MAP(i), and finally form a positive sample pair, that is, the positive sample pair is {f(i│θ Q ),f(MAP(i)|θ K )}.

[0116] In this embodiment, the scene consistency positive sample selection strategy adopted can select more accurate positive samples. The contrastive learning branch of the twin network is then used in combination with the contrastive loss function to pre-train the shot feature extractor on unlabeled video data, thereby effectively improving the generalization ability of the trained shot feature extractor, so that the pre-trained shot feature extractor has good performance in the shot scene segmentation task.

[0117] In one embodiment, the step of obtaining negative sample features includes:

[0118] When training the two-way model for the first time, the negative sample queue is initialized to obtain negative sample features;

[0119] When this is not the first time training the two-way model, the second shot features extracted by the second encoder in the current batch are cached in the negative sample queue; and during the next training, the second shot features in the negative sample queue are used as the negative sample features in the next training process.

[0120] The negative sample queue refers to a queue for storing negative samples. During each optimization of the two-way model, the server can obtain the corresponding negative samples from the negative sample queue.

[0121] Specifically, when the server trains the two-way model for the first time, the server can initialize the negative sample queue to obtain the initialized negative sample features; when the server is not training the two-way model for the first time, the server can cache the second shot features extracted by the current batch of the second encoder into the negative sample queue, and during the next training, the server obtains the cached historical shot features from the negative sample queue as the negative sample features in the next training process.

[0122] For example, for negative samples, a fixed-length first-in-first-out queue Q can be used in the embodiment of the present application to cache historical samples generated by the second encoder, that is, the samples in the queue Q are used as negative samples in each optimization process. When the server optimizes the first encoder and the second encoder in the two-way model for the first time, the server can directly obtain the negative sample features randomly generated when the queue Q is initialized, and use the above-obtained negative sample features as the negative sample features used for the first optimization. The negative sample features randomly generated when the queue Q is initialized are vectors composed of random numbers that obey a Gaussian distribution, that is, vectors composed of random numbers that obey a Gaussian distribution with a mean of 0 and a variance of 1.

[0123] In this embodiment, the historical samples generated by the second encoder, i.e., the key-value encoder, are used as negative samples. This can effectively increase the number of negative samples while ensuring that the video memory consumption remains unchanged, thereby effectively improving the generalization ability of the trained shot feature extractor, so that the pre-trained shot feature extractor has good performance in the shot scene segmentation task.

[0124] In one embodiment, the steps of extracting a third shot feature from a target video shot sequence using a shot feature extraction model and training a scene segmentation model based on the third shot feature include:

[0125] Obtain target video shot sequence and scene boundary labels;

[0126] Extract features of the target video shot sequence using a shot feature extraction model to obtain a third shot feature corresponding to the target video shot sequence;

[0127] Each third shot feature is input into a scene segmentation model for training to obtain a trained scene segmentation model.

[0128] The scene boundary labels refer to the corresponding scene boundary labels obtained by pre-labeling the target video shot sequence. That is, the target video shot sequence used to train the scene segmentation model in this application is a labeled video shot sequence.

[0129] Specifically, after the server uses the optimized first encoder as a shot feature extraction model, the server can obtain the target video shot sequence and scene boundary labels, and perform feature extraction on the target video shot sequence through the shot feature extraction model to obtain the third shot features corresponding to the target video shot sequence. Furthermore, the server can input the extracted third shot features into the scene segmentation model for training to obtain a trained scene segmentation model.

[0130] During the training phase of the video scene segmentation model, the goal is to perform supervised training on the video scene segmentation model. In this phase, the parameters of the lens feature extraction model trained in the previous phase are fixed. The server first inputs the labeled video lens sequence into the lens feature extraction model, so that the corresponding embedded features can be extracted using the trained lens feature extraction model with fixed parameters. The server then inputs the extracted embedded features into the designed scene segmentation model, and the corresponding output sequence represents the scene boundary label prediction sequence. At the same time, during the training phase of the video scene segmentation model, the server uses the cross-entropy loss function as the optimization target to train the scene segmentation model. Finally, the gradient backpropagation strategy is used to update the network parameters of the video scene segmentation model. The hardware environment of the entire training process relies on a server with high-performance computing capabilities equipped with a GPU.

[0131] For example, during the training phase of the video scene segmentation model, the server can use Bi-LSTM and three fully connected layers as the basic network framework of the scene segmentation model, and use the SGD optimizer for optimization, with the initial learning rate set to 0.1, the training batch size set to 32, and the training cycle set to 200.

[0132] In this embodiment, it is proposed to use a Bi-LSTM sequence modeling method to classify the scene boundaries of each shot, thereby effectively reducing the inductive bias of the scene boundary model on the number of scene boundary shots, showing better generalization ability, and being able to accurately and efficiently segment video scenes, greatly improving the performance of large-scale supervised algorithm frameworks, and solving problems such as poor generalization ability of scene segmentation models and poor accuracy of video scene segmentation.

[0133] In one embodiment, the step of performing video scene segmentation on the video to be segmented based on the trained scene segmentation model includes:

[0134] Get the video to be segmented;

[0135] Detecting the shot boundaries of the video to be segmented to obtain the video shots of the video to be segmented;

[0136] Extracting a fourth shot feature from the video shot of the video to be segmented;

[0137] The trained scene segmentation model is used to identify scene boundaries of the fourth shot feature to obtain each video scene of the video to be segmented.

[0138] Among them, the shot boundary refers to the boundary frame corresponding to a shot, that is, detecting whether a video frame in the video is a shot boundary frame. The shot boundary frame can include the boundary start frame and boundary end frame of each shot. For example, the video frames contained in each shot in the segmented video are detected to obtain the boundary start frame sequence number and boundary end frame sequence number of each shot.

[0139] The fourth shot feature refers to the shot feature obtained by extracting features from the video to be segmented. For example, the shot feature obtained by extracting features from any video to be segmented can be called the fourth shot feature.

[0140] Scene boundary recognition involves identifying a shot feature's scene boundary, determining whether it represents a scene boundary. The resulting judgment results can range from scene boundary to non-scene boundary. For example, if a shot feature is determined to represent a scene boundary, a corresponding prediction value of 1 may be output; if it is determined to represent a non-scene boundary, a corresponding prediction value of 0 may be output.

[0141] Specifically, the server extracts a third shot feature from a target video shot sequence through a shot feature extraction model, and trains a scene segmentation model based on the third shot feature. After obtaining the trained scene segmentation model, the server can obtain a video to be segmented, which contains at least two shots. The server can detect shot boundaries in the video to be segmented to obtain each video shot in the video to be segmented, that is, the server can detect each video frame in the video to be segmented based on a preset boundary detection method, that is, the server detects whether each video frame in the video to be segmented is a shot boundary frame. The shot boundary frame can include a boundary start frame and a boundary end frame of each shot. When the server detects that a video frame is a boundary start frame of a shot, the serial number of the video frame is recorded as the boundary start frame serial number of the shot; when the server detects that a video frame is a boundary end frame of a shot, the serial number of the video frame is recorded as the boundary end frame serial number of the shot. In this way, the server can obtain the boundaries of each video shot in the video to be segmented. Furthermore, the server may extract a fourth shot feature from the video shot of the video to be segmented, and perform scene boundary recognition on the fourth shot feature using a trained scene segmentation model to obtain each video scene of the video to be segmented.

[0142] For example, assuming that the video A to be segmented is: shot 1 → shot 2 → shot 3, that is, the video to be segmented contains 3 video shots, the server can detect whether each video frame in the video to be segmented is a shot boundary frame based on a preset boundary detection method, and obtain the boundary start frame number and boundary end frame number of each shot in the video to be segmented. Furthermore, the server can input the first frame corresponding to the boundary start frame number of each shot in the video to be segmented, the last frame corresponding to the boundary end frame number, and the intermediate frames into the shot feature extraction model to extract the fourth shot feature corresponding to each shot, and perform scene boundary recognition on the fourth shot feature through the trained scene segmentation model to obtain each video scene of the video to be segmented. For example, the server inputs the fourth shot feature into a trained scene segmentation model to identify scene boundaries. The resulting output sequence A represents a predicted sequence of scene boundary labels. Assuming output sequence A is 110, this indicates that shot 1 and shot 2 in video A to be segmented are scene boundaries, and shot 3 is not a scene boundary. After video scene segmentation, video A to be segmented is: shot 1 is one video scene, and shots 2 to 3 are another video scene. In other words, video A to be segmented contains two video scenes. This allows for accurate and efficient video scene segmentation in any video, significantly improving the performance of large-scale supervised algorithm frameworks.

[0143] In this application, when performing shot boundary detection on the original video to be segmented, various shot boundary detection methods may be employed, which are not specifically limited herein. Furthermore, when extracting fourth shot features from the video shots of the video to be segmented, this application includes, but is not limited to, extracting features from the video shots of the video to be segmented using a trained shot feature extraction model, and other feature extraction methods may also be employed.

[0144] In addition, in the embodiments of the present application, when extracting lens features, all video frames in the lens can be used as input data to extract corresponding lens features, or three video frame images of the first frame, the last frame and the middle frame in each lens can be selected as input data to extract corresponding lens features. Custom settings can be made according to the different needs of users using the lens feature extraction model.

[0145] This application also provides an application scenario, which applies the above-mentioned video scene segmentation method. Specifically, the application of the video scene segmentation method in this application scenario is as follows:

[0146] When a long video needs to be segmented, the aforementioned video scene segmentation method can be used. This method uses the trained scene segmentation model to automatically segment the video into several independent sub-videos in the temporal domain based on the different scenes within the video. Furthermore, the aforementioned video scene segmentation method can also be applied to video shot retrieval. For a given shot, similar scenes can be automatically retrieved and sorted by similarity.

[0147] The method provided in the embodiment of the present application can be applied to the scenario of the long video scene segmentation task. The following takes the long video scene segmentation task scenario as an example to illustrate the video scene segmentation method provided in the embodiment of the present application.

[0148] Video scene segmentation is an important technology in video understanding. It mainly uses a long video as a segmentation clue according to the different scene contents, and divides the long video into several independent sub-videos based on the shot dimension. For this technology, extracting good shot features and selecting an appropriate shot segmentation model are crucial.

[0149] Traditionally, shot feature extraction schemes are mainly divided into two categories. One is to use a model pre-trained on a video classification dataset to extract shot features. The disadvantage is that this type of method requires a large amount of labeled data for pre-training and has task knowledge bias. The other is to use a self-supervised algorithm to pre-train the shot feature extractor on unlabeled data. However, the shot feature extractors trained by the self-supervised algorithm based on image classification tasks still have poor generalization capabilities in video scene segmentation tasks.

[0150] Traditionally, the video scene segmentation algorithm mainly includes two steps. The first step is to use a shot feature extractor to extract shot features, and the second step is to use the extracted shot features to model and optimize the scene segmentation problem. For the shot feature extraction step, most algorithms rely on models based on supervised pre-training to extract shot features. Other self-supervised algorithms mainly select the two most adjacent shot samples as positive samples according to the Euclidean distance, and other historical samples as negative samples. In the subsequent optimization process, the positive sample pairs are continuously moved closer to each other in spatial distance, and the negative sample pairs are moved further away from each other. For the scene segmentation step, the core idea of ​​the traditional algorithm is to convert the task into a binary classification task of shot boundaries. Specifically, the positive sample represents that the boundary is a scene boundary, and the negative sample represents a non-scene boundary.

[0151] Although existing deep learning-based video scene segmentation algorithms utilize extracted shot features to model shot boundaries and have achieved some success, the shot feature extractors derived from supervised pre-training methods require a large amount of labeled data. Furthermore, these models are typically trained on video classification data, which can lead to task knowledge bias and poor generalization in scene segmentation tasks. While there are unsupervised approaches to pre-training shot feature extractors, such as optimizing by sampling positive pairs using nearest neighbor selection, these improvements remain limited. On the other hand, the current scene segmentation modeling methods all aggregate a number of adjacent lens features on both sides of a certain lens boundary to obtain the features of the lens boundary, and then further classify the features of the lens boundary to decide whether the boundary is a scene boundary. The problem brought about by this is that the number of lens features aggregated at each lens boundary is not easy to determine, and the number of lenses each scene has is also not fixed. The number of lenses is usually determined by hyperparameter sensitivity testing, so it will introduce a strong inductive bias, making the model training more difficult and the generalization ability poor, which can easily lead to poor generalization ability of the video scene segmentation model, resulting in poor accuracy of video scene segmentation.

[0152] Therefore, in order to solve the above problems, the present application provides a consistent representation learning method for video scene segmentation, proposes a self-supervised pre-training algorithm framework based on scene consistency, and pre-trains the shot feature extractor on unlabeled video data. This framework mainly includes the use of a positive sample selection method based on scene consistency selection, and at the same time uses a data augmentation method of shuffling scene fragments to improve the diversity of shots in the training batch. Finally, the proposed scene boundary-independent model is combined to complete the video scene segmentation task. For the shot feature extractor, a designed unsupervised pre-training algorithm is used to pre-train the shot feature extractor using unlabeled video data. The pre-trained shot feature extractor has good performance in the shot scene segmentation task. At the same time, a scene boundary free (Boundaryfree) model is proposed to model and optimize the video scene segmentation task, that is, it is proposed to use a Bi-LSTM sequence modeling method to classify the scene boundary of each shot, thereby effectively reducing the inductive bias of the scene boundary model on the number of scene boundary shots, showing better generalization ability, and being able to accurately and efficiently segment video scenes, greatly improving the performance of large-scale supervised algorithm frameworks, and solving problems such as poor generalization ability of scene segmentation models and poor accuracy of video scene segmentation.

[0153] like Figure 5 The figure shows a comparison diagram of the scene consistency selection method and the nearest neighbor selection method in the video scene segmentation task. Figure 5 The nearest neighbor selection method used in the video clip under the lens dimension in (a) is a processing method commonly used in traditional methods; Figure 5 The scene consistency selection method used in the video clip under the scene dimension (b) is the processing method proposed and adopted in this application.

[0154] In the video scene segmentation task, because it is necessary to judge the boundary of each shot contained in the video, that is, to judge whether the shot boundary is the scene boundary, the following method is used: Figure 5 The nearest neighbor method shown in (a) selects several consecutive shot features for aggregation and only determines whether the most central boundary is the scene boundary. Experiments show that the model effect is very sensitive to the number of consecutive shots, and the use of Figure 5 The nearest neighbor method shown in (a) uses experiments to select the number of shots, so there is an inductive bias, that is, the number of shots on both sides of the shot boundary to be decided is determined by experimental induction.

[0155] The examples proposed in this application are as follows Figure 5The scene consistency positive sample selection strategy used in the video clips under the scene dimension (b) shown in the figure can select more accurate positive samples. The contrastive learning branch of the twin network is then used in combination with the contrastive loss function to pre-train the shot feature extractor on unlabeled video data, thereby effectively improving the generalization ability of the trained shot feature extractor, so that the pre-trained shot feature extractor has good performance in the shot scene segmentation task. At the same time, a scene boundary free model is proposed to model and optimize the video scene segmentation task, that is, it is proposed to use a Bi-LSTM sequence modeling method to classify the scene boundary for each shot, thereby effectively reducing the inductive bias of the scene boundary model on the number of scene boundary shots, showing better generalization ability, and being able to accurately and efficiently segment the video scene, greatly improving the performance of the large-scale supervised algorithm framework, and solving the problems of poor generalization ability of the scene segmentation model and poor accuracy of video scene segmentation.

[0156] On the product side, the method provided in the embodiments of the present application can achieve the following functions:

[0157] The consistent representation learning method for video scene segmentation provided in the embodiments of this application can be applied to the task of long video scene segmentation. This is usually defined on the product side as a video stripping capability. That is, for a given long video, the method provided in the embodiments of this application can automatically split the video into several independent sub-videos in the time domain based on the different scenes in the video. In addition, the consistent representation learning method for video scene segmentation provided in the embodiments of this application can also be applied to video shot retrieval. That is, for a given shot, shots of similar scenes are automatically retrieved and sorted by similarity.

[0158] On the technical side, Figure 6 As shown in the figure, it is a schematic diagram of the overall framework of the consistent representation learning method for video scene segmentation. Figure 6 The algorithm framework shown mainly consists of two stages:

[0159] (1) Shot representation learning stage. Figure 6In the (a) shot representation learning stage shown in the figure, a dual-path training branch consisting of a key encoder and a query encoder is used, which is usually called a twin network. For the input shot sequence, the Aug_K and Aug_Q data augmentation methods are first used to obtain the enhanced input shot sequence. Then, in the query encoder, the corresponding positive samples are screened out for each shot in the input shot sequence, and based on the screened positive samples, an index mapping function MAP(i) is obtained. MAP(i) represents the positive sample index corresponding to sample i. Then, the positive sample actually used for training is selected from the key encoder according to the index provided by MAP(i). Finally, the entire pre-training process is optimized by minimizing the designed contrast loss function. After the training is completed, the query encoder in the dual-path training branch is used as the shot feature extractor M used in the subsequent stage.

[0160] In the embodiment of the present application, the query encoder is updated by gradient, and the key-value encoder is updated by momentum. The purpose is to increase the number of negative samples in disguise, which is equivalent to saving video memory and increasing the training batch size. Therefore, the key-value encoder has hysteresis, and the query encoder can better reflect the current state of feature learning. Therefore, the query encoder is ultimately used as the shot feature extractor M.

[0161] (2) Video scene segmentation stage. Figure 6 In the (b) video scene segmentation stage shown in the figure, the fixed-parameter query encoder, namely the shot feature extractor M, trained in the (a) shot representation learning stage, is used to extract features from the input video shot sequence. The extracted features are then input into the designed MLP / Bi-LSTM-based video scene segmentation model D to complete the task of this stage.

[0162] In the inference stage, for any input video shot sequence S_t, the corresponding shot features are extracted by the shot feature extractor M, and then the extracted shot features are input into the scene segmentation model D to predict the scene segmentation label sequence Y_t.

[0163] The consistent representation learning method for video scene segmentation proposed in this application includes two tasks or steps:

[0164] 1. Pre-train shot feature extractor M;

[0165] 2. Use the shot features extracted by the trained shot feature extractor M to train the scene segmentation model D.

[0166] For the pre-training task M of training the shot feature extractor, given an unlabeled shot sequence, the feature extractor M is pre-trained on a proxy task. The proxy task is to continuously narrow the representation distance between positive samples and widen the representation distance between negative samples during the pre-training process. The positive sample selection scheme based on scene consistency proposed in the embodiment of the present application will effectively improve the generalization ability of the shot feature extractor M. The specific positive and negative sample selection and proxy task optimization are introduced in detail below.

[0167] 1.1 Siamese Network Structure and Data Enhancement

[0168] In order to pre-train the shot extractor M on unlabeled data, this scheme uses a twin network structure to extract shot features and generate positive and negative sample pairs, such as Figure 6 Specifically, in the training preparation phase, the model structures and initial parameters of the key encoder and query encoder are exactly the same. Subsequently, query samples are first obtained in the query encoder branch, and positive samples are selected to obtain the mapping relationship function MAP(i). Then, based on the mapping relationship function MAP(i), the samples generated in the key encoder branch are re-indexed to obtain key samples, and then positive sample pairs are obtained. The twin network structure model is optimized and updated by minimizing the contrast loss function, where the query encoder is updated by gradient and the key encoder is updated by momentum.

[0169] In the embodiment of the present application, the query encoder adopts a gradient update method to reflect the current learning state, and the key-value encoder adopts a momentum update method to increase the number of negative samples and save video memory.

[0170] In contrastive learning, due to the dual-branch structure of the Siamese network, for the same sample input, two views are obtained after random data augmentation, and these are input into the dual-branch to produce two representations of the sample. During training, a consistency constraint loss, i.e., the contrastive learning loss function, is used to gradually bring the two representations closer in spatial distribution. This makes the Siamese network insensitive to various transformations of the same sample, and allows it to produce relatively consistent representations.

[0171] During the pre-training M phase of the pre-trained shot feature extractor, the goal of "narrowing the representational distance between positive samples and widening the representational distance between negative samples" is to continuously repeat this process during the model's learning process. Ultimately, the model can make the feature distributions of similar samples closer in the high-dimensional feature space, while dissimilar feature samples are further apart. That is, after pre-training, the shot feature extraction model possesses basic feature semantic similarity discrimination capabilities. The shot feature extractor M in the embodiment of this application is a neural network based on the ResNet50 structure. Using ResNet50 as the underlying feature extractor structure has the following advantages: 1. Simple structure, fast speed, and easy deployment; 2. Compliance with the principle of fairness, facilitating comparison with other methods. It will be understood that the embodiment of this application actually operates on embedded features and is therefore not dependent on a specific network structure. The shot feature extractor M can also use other network model structures.

[0172] In addition, data augmentation is crucial in the shot representation learning stage. The data augmentation used in this framework is mainly divided into two types: one is to regularly shuffle the input shot sequence, which is called scene-independent shot sequence shuffling here, and the other is to asymmetric image enhancement of individual shots in the input shot sequence. In the scene-independent shot sequence shuffling method, in order to improve the richness of shots and scenes in a batch, this framework proposes to use continuous shots of a fixed length of ρ as independent units to shuffle the entire shot sequence, such as Figure 4 , which is a schematic diagram of a shot-independent shot sequence scrambling method.

[0173] After obtaining the shuffled input shot sequence, random data augmentation is performed on each shot in the shuffled input shot sequence. This framework uses an asymmetric data augmentation method. Specifically, the data augmentation scheme used in the query encoder branch, namely Aug_Q, includes random cropping, Gaussian blurring, and random horizontal inversion operations, while the data augmentation scheme used in the key-value encoder branch, namely Aug_K, includes random scaling, random cropping, random color transformation, random grayscale transformation, Gaussian blurring, and random horizontal inversion operations. Because using the same data augmentation for both the query encoder and the key-value encoder can reduce the stability of model learning and the ultimate generalization, in this embodiment of the application, different data augmentation methods are used for the input shot sequences used in the query encoder branch and the key-value encoder branch.

[0174] 1.2 Positive and negative sample selection based on scene consistency

[0175] The selection of positive and negative samples is the core strategy of the shot representation learning stage. This framework adopts a positive sample selection strategy based on scene consistency. Let the query encoder be f(·|θ Q ), the key-value encoder is f(·|θK ), where θ Q and θ K In order to obtain the network parameters, this scheme first uses the K-Means algorithm to cluster the shot features extracted by the query encoder, and obtains several clusters and the central sample subscript I of each cluster. C ={i c1 ,i c2 ,i c3 ,…,i (c#class)}, where #class is the number of clusters. For the i-th sample in each cluster, the center sample of the cluster is selected as the corresponding positive sample, and its index value in the batch is recorded as the mapping function MAP(i). The actual positive sample is obtained in the key-value encoder through MAP(i) to form a positive sample pair, that is, the positive sample pair is {f(i│θ Q ),f(MAP(i)|θ K The re-indexing method is adopted in the embodiment of the present application to facilitate the unified representation of mathematical modeling, that is, different positive and negative sample strategies are represented by different mapping relationship functions MAP.

[0176] For negative samples, this framework uses a fixed-length, first-in, first-out queue to cache historical samples generated by the key-value encoder. The samples in this queue are used as negative samples in each optimization process. The reason why historical samples generated by the key-value encoder are used as negative samples in this application is that the key-value encoder uses momentum updates. Therefore, the key-value encoder parameter updates are slow, and the model parameters of samples in adjacent batches remain basically unchanged. This is similar to increasing the number of negative samples while keeping the video memory consumption unchanged.

[0177] In one embodiment, the mapping function MAP(i) obtained by the positive sample selection method based on scene consistency is shown in the following formula (1):

[0178]

[0179] Where X in formula (1) is the input shot sequence data; ||f(X[i]|θ Q )-f(X[j]|θ Q )||2 means that in the query encoder, the Euclidean distance between the representations of sample X[i] and sample X[j] is calculated. The overall meaning of formula (1) is to find the cluster center subscript closest to each sample.

[0180] For example, a group of samples X undergoes dual-path training branches, i.e., the key-value encoder branch generates X1 and the query encoder branch generates X2. For each pair of samples X1[i] and X2[i], we tentatively treat them as positive sample pairs, i.e., the samples with the same subscripts in the two groups of samples are regarded as positive sample pairs. However, they actually come from the same sample (shot), so it is necessary to reorganize this matching relationship. That is, the re-indexing method is adopted in this application, the purpose of which is to rearrange X1 so that the sample pairs with the same subscripts of X1 and X2 constitute positive sample pairs. Before re-indexing, the X2 samples generated by the query encoder branch are clustered, and new samples X3 are obtained according to the mapping relationship from subscript to subscript in the mapping function MAP(i). The mapping function MAP(i) indicates that in X1, the i-th sample is rearranged as sample X1[MAP(i)], which is equivalent to recombining a new set of samples X3. It can be understood that in the new sample X3, X3[i]=X1[MAP[i]], and X3[i] and X2[i] are positive sample pairs.

[0181] 1.3 Loss Function Design

[0182] After obtaining the positive and negative sample pairs, the designed contrast loss is used to optimize the shot feature extraction model. The contrast loss function is shown in the following formula (2):

[0183]

[0184] Where q in formula (2) is the query sample, k + is the positive sample set of sample q, k - is the set of negative samples of sample q, sim(·,·) is the cosine similarity between two vectors, τ is the temperature parameter, and e is a natural constant. The temperature parameter τ is a pre-set fixed value that can be set to 0.07.

[0185] 2. Use the shot features extracted by the trained shot feature extractor M to train the scene segmentation model D.

[0186] For the task of training a scene segmentation model D, given a video shot sequence S_t (t = 0, 1, 2, 3, ..., T), the goal is to solve an output sequence Y_t (t = 0, 1, 2, 3, ..., T), the value of which represents the scene segmentation label. The scene segmentation model D proposed in this application can achieve better performance while reducing the number of parameters. Among them, the scene segmentation model D proposed in this application is a neural network based on the Bi-LSTM structure.

[0187] like Figure 7 The figure shows a comparison diagram of the scene boundary model and the scene free model.

[0188] Different from the traditional scene boundary based model, this application proposes a scene boundary free model to model and optimize the video scene segmentation task. Figure 7 (b) Scene boundary free model shown in . Taking data with an input length of B*Shot-Len*N as an example, B is the training batch size, Shot-Len is the number of shots processed in a single batch, and N is the dimension of the shot feature. The scene boundary model output in the traditional method is B*2, that is, the traditional method only models the center boundary of the shot. However, in the embodiment of the present application, it is proposed to use a Bi-LSTM sequence modeling method to classify the scene boundary of each shot in the input data. That is, after the Bi-LSTM model sequence modeling, the sequence output is B*Shot-Len*2. The advantages of this are: (1) it can effectively reduce the model complexity and parameter quantity; (2) improve the efficiency of model inference; (3) effectively reduce the model's inductive bias on the number of scene boundary shots.

[0189] like Figure 7 As shown, observing the output dimension, the prediction sequence dimension of the boundary-free model output is consistent with the number of input shots, as shown in Figure 7 As shown in (b) the scene boundary free model in the traditional method, the output of the boundary model method is always 1, that is, the traditional method only models the center boundary of the lens and only determines whether the center boundary of the lens is the scene boundary, such as Figure 7 The method proposed in this application, however, outputs predictions corresponding to each scene boundary, independent of the number of adjacent shots aggregated. Therefore, the method proposed in this application is a sequence-to-sequence modeling approach, offering more flexible temporal modeling capabilities.

[0190] 3. Network training strategy

[0191] The following is a brief description of the training strategy of this framework. This framework is mainly divided into two training stages, such as Figure 8 , which is a flowchart of the training strategy in the framework of the consistent representation learning method for video scene segmentation.

[0192] In such Figure 8In the training strategy diagram of this framework shown in the figure, in the first phase, shot representation learning, a scene consistency-based positive sample selection strategy and data augmentation strategy are adopted. The contrastive learning branch of the Siamese network is then used in conjunction with a contrastive loss function to pre-train the shot feature extractor on unlabeled video data. In the second phase, the video scene segmentation phase, the fixed-parameter shot feature extractor trained in the first phase is used to extract embedded features from the labeled input shot sequence. The scene boundary free model and scene boundary labels proposed in this framework are then used to train the scene segmentation model using the cross-entropy loss function. The entire training process relies on a high-performance computing server with a GPU.

[0193] The specific training steps are as follows:

[0194] 1) Shot representation learning stage

[0195] In the shot representation learning phase, the goal is to perform unsupervised pre-training of the shot feature extractor. This framework first performs data augmentation on the video shot sequence before inputting it into the twin network to extract shot features. A designed positive and negative sample selection strategy is then used to select positive and negative sample pairs, and a contrastive loss function is used as the optimization objective. Finally, gradient backpropagation is used to update the query encoder network parameters, and a momentum strategy is used to update the key-value encoder network parameters. In this phase, the ResNet50 network can be used as the basic network framework for the shot extractor, and the SGD optimizer is used for optimization. The initial learning rate is set to 0.03, the batch size in training is 1024, the training epochs are 100, the size of the negative sample queue is 65535, the temperature parameter is set to 0.07, and the momentum parameter used to update the key-value encoder is set to 0.999.

[0196] In this application, unsupervised pre-training is used for the shot feature extractor, mainly to reduce the labeling cost. A large amount of unlabeled data can be used for pre-training, with the goal of obtaining a good initial feature extractor. Through the set pre-training tasks, the shot feature extractor is trained, and the temporal and non-temporal relationship features inherent in the video are mined from a large amount of unlabeled video data. This makes the individual shot features extracted by the feature extractor have certain shot category discriminability and temporal relationship information. Then, in the video scene segmentation task, fine-tuning using a small amount of scene segmentation labeled data can show good generalization ability.

[0197] 2) Video scene segmentation stage

[0198] The goal of the video scene segmentation phase is to perform supervised training on the video scene segmentation model. In this phase, the parameters of the shot feature extractor trained in the previous phase are fixed. First, the labeled video shot sequence is extracted using the shot feature extractor to obtain the corresponding embedded features. This embedded feature is then input into the designed scene boundary free model. The resulting output sequence represents the scene boundary label prediction sequence. The cross-entropy loss function is used as the optimization objective, and finally, the gradient backpropagation strategy is used to update the network parameters. This phase uses a Bi-LSTM and three fully connected layers as the basic network framework of the scene segmentation model. The SGD optimizer is used for optimization, with an initial learning rate of 0.1, a training batch size of 32, and a training epoch of 200. Since we have already reached the downstream task modeling phase, i.e., the second phase, only a small amount of data is required to train the downstream video scene segmentation model.

[0199] In this embodiment, the shot feature extractor can be pre-trained on unlabeled video data, and combined with the designed boundary-free model, the video scenes in any video can be accurately and efficiently segmented, while also significantly improving the performance of large-scale supervised algorithm frameworks.

[0200] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0201] Based on the same inventive concept, embodiments of the present application also provide a video scene segmentation device for implementing the aforementioned video scene segmentation method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the one or more video scene segmentation device embodiments provided below can be found in the aforementioned limitations of the video scene segmentation method and will not be further elaborated here.

[0202] In one embodiment, Figure 9 As shown, a video scene segmentation device is provided, comprising: an extraction module 902, a determination module 904, an optimization module 906, a training module 908 and a segmentation module 910, wherein:

[0203] The extraction module 902 is configured to extract features from the video shot sequence using a first encoder and a second encoder of a two-way model to obtain first shot features and second shot features.

[0204] The determination module 904 is configured to determine a first sample feature in the first shot feature, and determine a second sample feature in the second shot feature based on the encoding of the first sample feature; the first sample feature and the second sample feature are used as positive sample features.

[0205] The optimization module 906 is used to obtain negative sample features and optimize the first encoder and the second encoder based on the loss value between the negative sample features and the positive sample features; wherein the optimized first encoder is used as the shot feature extraction model.

[0206] The training module 908 is configured to extract a third shot feature from the target video shot sequence using a shot feature extraction model, and train a scene segmentation model based on the third shot feature.

[0207] The segmentation module 910 is configured to perform video scene segmentation on the video to be segmented based on the trained scene segmentation model.

[0208] In one embodiment, the apparatus further includes: an acquisition module for acquiring temporally continuous video shots to obtain a continuous video shot sequence; a shuffling processing module for shuffling the video shots in the continuous video shot sequence to obtain a shuffled video shot sequence; and an enhancement processing module for performing data enhancement processing on each video shot in the shuffled video shot sequence to obtain a video shot sequence for feature extraction.

[0209] In one embodiment, the device further includes: a segmentation module for segmenting each video shot in the continuous video shot sequence into segments to obtain video segments; and a shuffling processing module for shuffling the video segments of each video shot in the continuous video shot sequence to obtain a shuffled video shot sequence.

[0210] In one embodiment, the apparatus further comprises: an acquisition module further configured to acquire at least one video shot to be inserted; and an insertion module configured to insert the at least one video shot to be inserted between adjacent video shots in the video shot sequence to obtain a disordered video shot sequence.

[0211] In one embodiment, the enhancement processing module is further used to perform a first data enhancement process on each video shot in the disordered video shot sequence to obtain a video shot sequence for input to the first encoder for feature extraction; the first data enhancement process includes at least one of cropping, blurring, and inversion operations; and perform a second data enhancement process on each video shot in the disordered video shot sequence to obtain a video shot sequence for input to the second encoder for feature extraction; the second data enhancement process includes at least one of scaling, cropping, pixel transformation, blurring, and horizontal inversion operations.

[0212] In one embodiment, the apparatus further includes: a clustering processing module, configured to perform clustering processing on the first lens features to obtain lens feature clusters and central lens features of each lens feature cluster; a selection module, configured to select the central lens features of each lens feature cluster as the first sample feature; and a recording module, configured to record the codes of the central lens features of each lens feature cluster.

[0213] In one embodiment, the determination module is further configured to use the code as a variable of an index mapping function; the index mapping function is configured to reflect a mapping relationship between a central lens feature of each lens feature cluster and a second sample feature; and based on the index mapping function, determine a second sample feature from the second lens feature.

[0214] In one embodiment, the apparatus further includes: an initialization module, configured to initialize a negative sample queue to obtain negative sample features when the two-way model is trained for the first time; a cache module, configured to cache the second shot features extracted by the second encoder in the current batch into the negative sample queue when the two-way model is not trained for the first time; and, during the next training, use the second shot features in the negative sample queue as negative sample features in the next training process.

[0215] In one embodiment, the acquisition module is further used to obtain a target video shot sequence and scene boundary labels; the extraction module is further used to perform feature extraction on the target video shot sequence using the shot feature extraction model to obtain third shot features corresponding to the target video shot sequence; and the training module is further used to input each of the third shot features into a scene segmentation model for training to obtain a trained scene segmentation model.

[0216] In one embodiment, the device further includes: an acquisition module for acquiring the video to be segmented; a detection module for detecting the shot boundaries of the video to be segmented to obtain the video shots of the video to be segmented; an extraction module for extracting fourth shot features from the video shots of the video to be segmented; and an identification module for performing scene boundary identification on the fourth shot features using the trained scene segmentation model to obtain the video scenes of the video to be segmented.

[0217] Each module in the above-mentioned video scene segmentation device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0218] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store segmentation data of the video scene. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video scene segmentation method is implemented.

[0219] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0220] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0221] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0222] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0223] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0224] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0225] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0226] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A video scene segmentation method, characterized in that: The method comprises: A first encoder and a second encoder of a dual-path model are used to extract features from a video shot sequence, respectively, to obtain first shot features and second shot features; the video shot sequence includes a first video shot sequence for input to the first encoder for feature extraction, and a second video shot sequence for input to the second encoder for feature extraction, wherein the first video shot sequence is obtained by performing cropping, blurring, and inverting operations on each video shot in a disordered video shot sequence, and the second video shot sequence is obtained by performing scaling, cropping, pixel transformation, blurring, and horizontal inversion operations on each video shot in the disordered video shot sequence; Clustering the first shot features to obtain shot feature clusters and central shot features of each of the lens feature clusters, selecting the central shot features of each of the lens feature clusters as first sample features, and determining second sample features from the second shot features based on the encoding of the first sample features; the first sample features and the second sample features are used as positive sample features; the encoding of the first sample features refers to identification information in the training batch to which each of the central shot features belongs; Obtaining negative sample features, and optimizing the first encoder and the second encoder based on a loss value between the negative sample features and the positive sample features; wherein the optimized first encoder serves as a shot feature extraction model; Extracting a third shot feature from a target video shot sequence using the shot feature extraction model, and training a scene segmentation model based on the third shot feature; The video scene is segmented based on the trained scene segmentation model.

2. The method according to claim 1, characterized in that Before extracting features from the video shot sequence using the first encoder and the second encoder of the two-way model respectively, the method further includes: Acquire temporally continuous video shots to obtain a continuous video shot sequence; Performing shuffling of the video shots in the continuous video shot sequence to obtain a shuffled video shot sequence; Data enhancement processing is performed on each video shot in the disordered video shot sequence to obtain a video shot sequence for feature extraction.

3. The method according to claim 2, characterized in that The step of performing random processing on the video shots in the continuous video shot sequence to obtain a random video shot sequence includes: Dividing each video shot in the continuous video shot sequence into segments to obtain video segments; The video segments of each video shot in the continuous video shot sequence are shuffled to obtain a shuffled video shot sequence.

4. The method according to claim 2, characterized in that The step of performing random processing on the video shots in the continuous video shot sequence to obtain a random video shot sequence includes: Obtain at least one video shot to be inserted; The at least one video shot to be inserted is inserted between adjacent video shots in the video shot sequence to obtain a disordered video shot sequence.

5. The method according to claim 1, wherein The method further includes: recording the code of the central shot feature of each shot feature cluster.

6. The method according to claim 5, characterized in that The determining a second sample feature in the second shot feature based on the encoding of the first sample feature includes: The code is used as a variable of an index mapping function; the index mapping function is used to reflect the mapping relationship between the central shot feature of each shot feature cluster and the second sample feature; Based on the index mapping function, a second sample feature is determined from the second lens feature.

7. The method according to claim 1, characterized in that The obtaining of negative sample features includes: When the two-way model is first trained, the negative sample queue is initialized to obtain negative sample features; When this is not the first time the two-way model is trained, the second shot features extracted by the second encoder in the current batch are cached in a negative sample queue; and during the next training, the second shot features in the negative sample queue are used as negative sample features in the next training process.

8. The method according to claim 1, characterized in that The step of extracting a third shot feature from a target video shot sequence by using the shot feature extraction model and training a scene segmentation model based on the third shot feature comprises: Obtain target video shot sequence and scene boundary labels; Extracting features from the target video shot sequence using the shot feature extraction model to obtain a third shot feature corresponding to the target video shot sequence; Each of the third shot features is input into a scene segmentation model for training to obtain a trained scene segmentation model.

9. The method according to claim 1, characterized in that The video scene segmentation based on the trained scene segmentation model includes: Get the video to be segmented; Detecting shot boundaries of the video to be segmented to obtain video shots of the video to be segmented; Extracting a fourth shot feature from the video shot of the video to be segmented; The trained scene segmentation model is used to perform scene boundary recognition on the fourth shot feature to obtain each video scene of the video to be segmented.

10. A video scene segmentation device, characterized in that: The device comprises: an extraction module configured to extract features from a video shot sequence using a first encoder and a second encoder of a dual-path model, respectively, to obtain first shot features and second shot features; the video shot sequence comprises a first video shot sequence for input to the first encoder for feature extraction, and a second video shot sequence for input to the second encoder for feature extraction, wherein the first video shot sequence is obtained by cropping, blurring, and reversing each video shot in a disordered video shot sequence, and the second video shot sequence is obtained by scaling, cropping, pixel transformation, blurring, and horizontal reversing each video shot in the disordered video shot sequence; a determination module configured to cluster the first shot features to obtain shot feature clusters and central shot features of each of the lens feature clusters, select the central shot features of each of the lens feature clusters as first sample features, and determine second sample features from the second shot features based on the encoding of the first sample features; the first sample features and the second sample features are used as positive sample features; the encoding of the first sample features refers to identification information in the training batch to which each of the central shot features belongs; an optimization module, configured to obtain negative sample features and optimize the first encoder and the second encoder based on a loss value between the negative sample features and the positive sample features; wherein the optimized first encoder serves as a shot feature extraction model; a training module, configured to extract a third shot feature from a target video shot sequence using the shot feature extraction model, and train a scene segmentation model based on the third shot feature; The segmentation module is used to perform video scene segmentation on the video to be segmented based on the trained scene segmentation model.

11. The video scene segmentation device according to claim 10, characterized in that: The device further comprises: An acquisition module is used to acquire temporally continuous video shots to obtain a continuous video shot sequence; a disorder processing module, configured to perform disorder processing on the video shots in the continuous video shot sequence to obtain a disordered video shot sequence; The enhancement processing module is used to perform data enhancement processing on each video shot in the disordered video shot sequence to obtain a video shot sequence for feature extraction.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images