Video clip extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a video segment extraction method and device, electronic equipment and a storage medium, belongs to the technical field of video processing, and is suitable for the financial and medical fields. The method comprises the following steps: acquiring an original video, and performing video frame extraction on the original video to obtain original video frames; performing time sequence feature extraction on the original video frames to obtain time sequence feature vectors; performing change weight extraction on the time sequence feature vectors to obtain change weights; performing picture change screening on the time sequence feature vectors according to the change weights to obtain screened feature vectors; performing subtitle-free frame detection on the screened feature vectors to obtain video frame identifiers; and performing segment clustering on the original video frames according to the video frame identifiers, the change weights and the screened feature vectors to obtain target video segments. The embodiment of the application can accurately detect subtitle-free video segments in a video.

Description

Technical Field

[0001] This application relates to the field of video processing technology, applicable to the financial and medical fields, and particularly to a video clip extraction method and apparatus, electronic device and storage medium. Background Technology

[0002] Video clips without subtitles refer to segments of video without subtitles. Identifying these clips can help deliver more targeted advertising content. For example, in insurance product advertisements, identifying subtitle-free clips allows for the precise insertion of ads without compromising the video's viewing experience. Similarly, in medical teaching scenarios, identifying subtitle-free clips allows for the accurate insertion of medical device names. Existing methods for detecting subtitle-free content include OCR to recognize text within video frames, but this accuracy is low due to the presence of text on road signs or billboards within the video frame. Another method uses neural networks to identify subtitle-free clips, but most methods rely on static frame analysis, which is less effective due to dynamic elements such as lighting changes and rapid transitions in videos. Therefore, accurately detecting subtitle-free video clips is a pressing issue that needs to be addressed. Summary of the Invention

[0003] The main objective of this application is to provide a video segment extraction method, apparatus, electronic device, and storage medium, which aims to accurately detect video segments without subtitles.

[0004] To achieve the above objectives, a first aspect of this application proposes a video segment extraction method, the method comprising: The original video is acquired, and video frames are extracted from the original video to obtain the original video frames; Temporal feature extraction is performed on the original video frames to obtain temporal feature vectors; The change weights are extracted from the time-series feature vectors to obtain the change weights. Based on the change weight, the temporal feature vector is filtered for image change to obtain the filtered feature vector; The filtered feature vectors are subjected to subtitle-free frame detection to obtain video frame identifiers; the video frame identifiers are used to indicate whether the original video frame contains or does not contain subtitles. The original video frames are clustered into segments based on the video frame identifier, the change weight, and the filtered feature vector to obtain target video segments; wherein, the target video segments do not contain subtitles.

[0005] In some embodiments, the step of extracting temporal features from the original video frames to obtain a temporal feature vector includes: Perform self-attention calculation on the original video frame to obtain at least one cross-window feature vector; Temporal fusion is performed on each of the cross-window feature vectors to obtain the temporal feature vector.

[0006] In some embodiments, the step of performing self-attention computation on the original video frame to obtain at least one cross-window feature vector includes: The original video frames are divided into matrix blocks to obtain at least one three-dimensional window block; Self-attention calculation is performed on each of the three-dimensional window blocks to obtain the in-window self-attention feature vector; Calculate the similarity between the self-attention feature vectors within the window to obtain the window similarity; The cross-window feature vector is obtained by weighted fusion of the window similarity and the in-window self-attention feature vector, and then the cross-window feature vector is added to a preset vector set. The cross-window feature vector is used as the original video frame. The original video frame is then divided into matrix blocks to obtain at least one three-dimensional window block until the first preset iteration condition is met. Each vector in the vector set is used as the cross-window feature vector to obtain at least one cross-window feature vector.

[0007] In some embodiments, the cross-window feature vectors have sequential identifiers; the temporal fusion of each of the cross-window feature vectors to obtain the temporal feature vector includes: Obtain the preset time-series state feature vector and the preset counter; Obtain the sequence identifier that is the same as the number in the counter to obtain the target identifier; The target cross-window vector is obtained from each of the cross-window feature vectors based on the target identifier; The target cross-window vector and the temporal state feature vector are fused in a temporal sequence to obtain a temporal fusion feature vector. The time-series fusion feature vector is updated according to the time-series state feature vector, the counter number is incremented according to the preset increment number, and the process of obtaining the sequence identifier that is the same as the counter number is returned to obtain the target identifier until the second preset iteration condition is reached. The time-series state feature vector is used as the time-series feature vector.

[0008] In some embodiments, the step of performing subtitle-free frame detection on the screened feature vector to obtain video frame identifiers includes: The screened feature vectors are divided into feature maps to obtain gated feature maps; The gated feature map is embedded to obtain an embedded graph vector; The gated feature map is embedded to obtain an embedded position vector; The embedding feature vector is obtained by summing the embedding graph vector and the embedding position vector. The embedded feature vector is subjected to subtitle-free frame detection to obtain the video frame identifier.

[0009] In some embodiments, the step of clustering the original video frames according to the video frame identifier, the change weight, and the filtered feature vector to obtain the target video segment includes: The edge weights are calculated based on the video frame identifier, the change weights, and the filtered feature vectors; wherein the edge weights are used to indicate the similarity between the original video frames. Based on the edge weights, graph clustering is performed on the original video frames to obtain preliminary subtitle-free video segments; The initial video segment without subtitles is subjected to non-maximum suppression processing to obtain the target video segment.

[0010] In some embodiments, the step of extracting video frames from the original video to obtain original video frames includes: The original video is decoded to obtain decoded video frames; The decoded video frames are scaled to obtain scaled video frames; The scaled video frame is pixel-normalized to obtain the original video frame.

[0011] To achieve the above objectives, a second aspect of this application provides a video segment extraction apparatus, the apparatus comprising: The data acquisition module is used to acquire the original video and extract video frames from the original video to obtain the original video frames. The feature extraction module is used to extract temporal features from the original video frames to obtain temporal feature vectors; The weight extraction module is used to extract the change weights from the time-series feature vector to obtain the change weights. The change amount filtering module is used to filter the temporal feature vector based on the change amount weight to obtain the filtered feature vector; The frame detection module is used to perform subtitle-free frame detection on the filtered feature vector to obtain a video frame identifier; the video frame identifier is used to indicate whether the original video frame contains subtitles or not. The segment clustering module is used to perform segment clustering on the original video frame based on the video frame identifier, the change weight, and the filtered feature vector to obtain target video segments; wherein, the target video segments do not contain subtitles.

[0012] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0014] This application proposes a video segment extraction method, apparatus, electronic device, and storage medium. It acquires the original video and extracts video frames to obtain original video frames. Next, it extracts temporal features from the original video frames to obtain temporal feature vectors. Then, it extracts change weights from the temporal feature vectors to obtain change weights, and filters the temporal feature vectors based on the change weights to obtain filtered feature vectors. Next, it uses the filtered feature vectors to detect frames without subtitles to obtain video frame identifiers. Finally, it clusters the original video frames based on the video frame identifiers, change weights, and filtered feature vectors to obtain target video segments. Thus, this application's embodiments, through temporal feature extraction and change weight extraction techniques, overcome the limitations of existing technologies that rely on static frame analysis. Furthermore, in dynamic scenes, it can better handle the impact of factors such as lighting changes and rapid transitions on detection accuracy, improving the detection accuracy of subtitle-free segments. Attached Figure Description

[0015] Figure 1 This is a flowchart of the video segment extraction method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S101 in the text; Figure 3 yes Figure 1 The flowchart of step S102 in the document; Figure 4 yes Figure 3 The flowchart of step S301 in the process; Figure 5 yes Figure 3 The flowchart of step S302 in the document; Figure 6 yes Figure 1 The flowchart of step S105 in the process; Figure 7 yes Figure 1 The flowchart of step S106 in the process; Figure 8 This is a schematic diagram of the video segment extraction device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] Video clips without subtitles refer to segments of video without any subtitles. Identifying these clips can help deliver more targeted advertising content. For example, in insurance product advertisements, identifying subtitle-free segments allows for precise ad insertion without compromising the video's viewing experience. Existing methods for detecting subtitle-free content include OCR to recognize text within video frames, but this accuracy is low due to the presence of text on road signs or billboards within the video frame. Another method uses neural networks to identify subtitle-free segments, but most methods rely on static frame analysis, which is less effective due to dynamic scenes such as lighting changes and rapid transitions in videos. Therefore, accurately detecting subtitle-free video clips is a pressing issue that needs to be addressed.

[0020] Based on this, embodiments of this application provide a video segment extraction method and apparatus, electronic device and storage medium, aimed at accurately detecting video segments without subtitles in a video.

[0021] This application provides a video segment extraction method, apparatus, electronic device, and storage medium, which are specifically described through the following embodiments. First, the video segment extraction method in this application is described.

[0022] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0023] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0024] The video segment extraction method provided in this application relates to the field of video processing technology and is applicable to the financial and medical fields. The video segment extraction method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the video segment extraction method, but is not limited to the above forms.

[0025] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0026] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0027] Figure 1 This is an optional flowchart of the video segment extraction method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0028] Step S101: Obtain the original video and extract video frames from the original video to obtain the original video frames; Step S102: Extract temporal features from the original video frames to obtain temporal feature vectors; Step S103: Extract the change weights from the time series feature vectors to obtain the change weights; Step S104: Filter the temporal feature vectors based on the weight of the change amount to obtain the filtered feature vectors. Step S105: Perform subtitle-free frame detection on the filtered feature vector to obtain video frame identifiers; the video frame identifiers are used to indicate whether the original video frame contains subtitles or not. Step S106: Cluster the original video frames according to the video frame identifier, change weight, and filtered feature vector to obtain the target video segment; wherein the target video segment does not contain subtitles.

[0029] Steps S101 to S106 of this embodiment involve acquiring the original video and extracting video frames to obtain original video frames. Next, temporal features are extracted from the original video frames to obtain temporal feature vectors. Then, change weights are extracted from the temporal feature vectors to obtain change weights, and the temporal feature vectors are filtered based on the change weights to obtain filtered feature vectors. Next, the filtered feature vectors are used to detect frames without subtitles to obtain video frame identifiers. Finally, the original video frames are clustered based on the video frame identifiers, change weights, and filtered feature vectors to obtain target video segments. Thus, this embodiment, through temporal feature extraction and change weight extraction techniques, overcomes the limitations of existing technologies that rely on static frame analysis. Furthermore, in dynamic scenes, it can better address the impact of factors such as lighting changes and rapid transitions on detection accuracy, improving the detection accuracy of subtitle-free segments.

[0030] Please see Figure 2 In some embodiments, step S101 may include, but is not limited to, steps S201 to S204: Step S201: Decode the original video data to obtain decoded video frames; Step S202: Scale the decoded video frames to obtain scaled video frames; Step S203: Normalize the pixels of the scaled video frame to obtain the original video frame.

[0031] Steps S201 to S204, as illustrated in this embodiment, involve decoding the original video to obtain decoded video frames; then, scaling the decoded video frames to obtain scaled video frames; and finally, pixel normalization of the scaled video frames to obtain the original video frames. In this way, this embodiment extracts image frames from the video through decoding, and then, by scaling and normalizing the image frames, ensures uniformity in the size and pixel values ​​of the input data, guaranteeing data format consistency in the input model and avoiding performance fluctuations caused by size and pixel differences.

[0032] In step S201 of some embodiments, data decoding refers to reading the original video file and loading it into memory using the OpenCV video processing library to obtain decoded video frames. The decoded image frames are typically image data in RGB or YUV format.

[0033] In step S202 of some embodiments, the size scaling process refers to scaling the decoded video frame to a size of 224×224 to obtain a scaled video frame.

[0034] In step S203 of some embodiments, pixel normalization processing refers to normalizing the image pixel values ​​to the range of [0,1], thereby eliminating the brightness differences between different video frames, avoiding algorithm performance fluctuations caused by differences in image pixel values, and thus improving the detection accuracy of video frames without subtitles.

[0035] Please see Figure 3 In some embodiments, step S102 may include, but is not limited to, steps S301 to S302: Step S301: Perform self-attention calculation on the original video frame to obtain at least one cross-window feature vector; Step S302: Perform temporal fusion on each cross-window feature vector to obtain a temporal feature vector.

[0036] Steps S301 to S302, as illustrated in this embodiment, involve performing self-attention calculation on the original video frames. This allows for the extraction of relevant features across different windows, resulting in at least one cross-window feature vector. This captures long-distance dependencies between different frames, thereby improving the accuracy and richness of feature extraction. Next, temporal fusion is performed on each cross-window feature vector to integrate information from the time series, forming a temporal feature vector. This ensures that the temporal dependencies of the video are maintained during frame processing, thus better reflecting the dynamic changes in the video.

[0037] Please see Figure 4 In some embodiments, step S301 may include, but is not limited to, steps S401 to S406: Step S401: Perform matrix segmentation on the original video frame to obtain at least one three-dimensional window block; Step S402: Perform self-attention calculation on each three-dimensional window block to obtain the self-attention feature vector within the window; Step S403: Calculate the similarity between self-attention feature vectors within the window to obtain the window similarity; Step S404: Perform weighted fusion based on window similarity and in-window self-attention feature vectors to obtain cross-window feature vectors, and add the cross-window feature vectors to a preset vector set; Step S405: Use the cross-window feature vector as the original video frame, return to perform matrix block division on the original video frame to obtain at least one three-dimensional window block, until the first preset iteration condition is reached. Step S406: Use each vector in the vector set as a cross-window feature vector to obtain at least one cross-window feature vector.

[0038] Steps S401 to S406 of this embodiment involve dividing the original video frame into multiple three-dimensional window blocks by matrix segmentation. A self-attention mechanism is used to independently calculate the intra-window self-attention feature vector for each three-dimensional window block, extracting spatial and temporal features within each block. After calculating the self-attention features within each three-dimensional window block, the similarity between the intra-window self-attention feature vectors is calculated to obtain window similarity. Then, a weighted fusion is performed based on the window similarity and the self-attention feature vectors to obtain a cross-window feature vector. Ultimately, this embodiment, through self-attention calculation and weighted fusion of the three-dimensional window blocks, enables each window block to be processed independently while preserving local features and strengthening its correlation with other window blocks.

[0039] It should be noted that steps S401 to S406 are processed by the 3D Swin Transformer backbone network. The 3D Swin Transformer includes multiple Swin3Dblocks, which are cascaded and connected.

[0040] In steps S401 to S406 of some embodiments, for the first Swin3Dblock in the 3D Swin Transformer backbone network, the original video frame is input into the Swin3Dblock for matrix segmentation to obtain multiple three-dimensional window blocks. For example, assuming the size of the original video frame is H×W×C (where H is the height, W is the width, and C is the number of channels), the original video frame is segmented according to the set window size (e.g., 4×4×4) to obtain multiple 4×4×4 three-dimensional window blocks, which is also a matrix.

[0041] By independently calculating the self-attention weights for each 3D window block using Swin3Dblock, an intra-window self-attention feature vector is obtained for each 3D window block. This independent calculation method enhances the information extraction capability of each window block within a specific local region. For example, in a video clip, different regions and time frames may contain different types of motion or change. Through self-attention calculation, each 3D window block can be weighted according to its specific internal patterns, thereby better capturing local spatial and temporal information.

[0042] The similarity between the self-attention feature vectors within the window is calculated, and the window similarity is shown in Equation (1): (1), in, It is the in-window self-attention feature vector. It refers to the first The first window and the first Similarity between windows.

[0043] The weighted fusion is performed based on window similarity and in-window self-attention feature vectors as shown in Equation (2): (2), in, This is a cross-window feature vector. This refers to the number of windows. For example, consider three 3D window blocks, whose corresponding in-window self-attention feature vectors are respectively... The similarity between the first window and the second window is The similarity between the second and the first is Similarly, there are a total of So, for the first window, its corresponding... for Similarly, we can obtain the cross-window feature vectors for the second window and the third window.

[0044] Then, the first, second, and third cross-window feature vectors are labeled with the corresponding layer, for example, the cross-window feature vectors of the first layer Swin3Dblock are added to the vector set.

[0045] Then, these cross-window feature vectors are spliced ​​back to their original positions in the video frame according to their corresponding 3D window blocks, replacing the original video frame, and input into the next layer of Swin3Dblock. This process is repeated until the cross-window feature vectors of each layer of Swin3Dblock are obtained, which means the first preset iteration condition is met. Finally, all vectors in the vector set are used as cross-window feature vectors.

[0046] On the other hand, this embodiment of the application achieves progressive extraction from shallow to deep features by cascading processing among multiple Swin3Dblocks. In each Swin3Dblock layer, the original video frame undergoes matrix partitioning, in-window self-attention calculation, and cross-window feature fusion to generate a cross-window feature vector, which is then passed to the next Swin3Dblock layer for further processing. Through this hierarchical structure, the model can progressively integrate more local information, strengthening the expressive power of features layer by layer.

[0047] Please see Figure 5 In some embodiments, the cross-window feature vectors have sequential identifiers, and step S302 includes, but is not limited to, steps S501 to S506: Step S501: Obtain the preset time-series state feature vector and the preset counter; Step S502: Obtain the sequence identifier that is the same as the number on the counter to obtain the target identifier; Step S503: Obtain the target cross-window vector from each cross-window feature vector based on the target identifier; Step S504: The target cross-window vector and the temporal state feature vector are fused in a temporal sequence to obtain a temporal fused feature vector; Step S505: Update the time-series state feature vector according to the time-series fusion feature vector, increment the counter according to the preset increment number, return to obtain the sequence identifier that is the same as the counter number, and obtain the target identifier until the second preset iteration condition is reached. Step S506: Use the time-series state feature vector as the time-series feature vector.

[0048] Steps S501 to S506 of this embodiment illustrate the following steps: A preset temporal state feature vector and a preset counter are obtained; a sequence identifier identical to the counter's value is obtained to acquire a target identifier; a target cross-window vector is obtained from each cross-window feature vector based on the target identifier; the target cross-window vector and the temporal state feature vector are temporally fused to obtain a temporally fused feature vector; the temporal state feature vector is updated based on the temporally fused feature vector; the counter's value is incremented according to a preset increment number; the process continues until a second preset iteration condition is met; the temporal state feature vector is used as the temporal feature vector. Thus, this embodiment uses the order between windows as temporal information and fuses each cross-window feature vector as temporal information to obtain a temporal feature vector representing a video frame.

[0049] It should be noted that steps S501 to S506 are time-series fusion performed by an LSTM network.

[0050] In steps S501 to S506 of some embodiments, firstly, a preset temporal state feature vector is obtained, which can be initialized. The counter starts from 1 (0 in binary in a computer, used to represent the first layer of Swing3Dblock). Each Swing3Dblock has a sequence identifier; the Swing3Dblock corresponding to the counter is obtained, and then the target cross-window vector corresponding to that Swing3Dblock is obtained. For example, the first layer of Swing3Dblock has three target cross-window vectors. .

[0051] The target cross-window vector and the temporal state feature vector are fused temporally to obtain the temporal fused feature vector. For example, the temporal state feature vector... and Input them together into the LSTM to obtain , and Input into LSTM, get ,Will and Input into LSTM, get ,but This is the time-series feature vector corresponding to a counter of 1.

[0052] Then, the speedometer is incremented by 1, and the process of calculating the temporal feature vector of the second-layer Swin3Dblock is returned. This is the temporal state feature vector. When the counter is 2 Input them together into the LSTM to obtain , When the counter is 2 Input into LSTM, get ,Will When the counter is 2 Input into LSTM, get ,but This is the time-series feature vector corresponding to a counter of 2.

[0053] It should be noted that in each layer, the temporal state feature vector All of them are the same as the preset temporal state feature vector obtained in step S501, and do not change with the traversal of each layer of Swin3Dblock.

[0054] The process continues until all Swing3Dblocks have been traversed, reaching the second preset iteration condition.

[0055] Finally, all layers corresponding to As temporal feature vectors, a vector set is obtained, the length of which is the same as that of the Swin3Dblock, denoted as . ,in is the length of the vector set.

[0056] In step S103 of some embodiments, the change weight extraction is generated by motion consistency analysis and implemented by the motion conditional attention gating module (MCAM). The specific calculation in the motion conditional attention gating module is shown in equation (3): (3), in, Indicates the first The weights of the changes in each time series feature vector. This is the preset sliding window size, which defaults to 8. It is the sigmoid activation function. This is a learnable weighted network for the motion-conditional attention gating module. Indicates the first Each time-varying feature vector. The L2 norm is used to measure temporal continuity and adaptively suppress regions that do not conform to motion consistency (such as flash transitions), enabling analysis from static frames to dynamic frames.

[0057] In step S104 of some embodiments, the image change amount filtering is as shown in equation (4): (4), in, This is the feature vector after screening. Indicates the first Each time-varying feature vector.

[0058] Through steps S103 to S104, motion consistency analysis and motion conditional attention gating modules are used to accurately extract motion change information from the video, and irrelevant dynamic changes (such as transitions or lighting changes) are effectively suppressed by filtering the change amount weights. This improves the detection accuracy of subsequent video detection without subtitles.

[0059] Please see Figure 6 In some embodiments, step S105 includes, but is not limited to, steps S601 to S605: Step S601: Divide the sieved feature vectors into feature maps to obtain gated feature maps; Step S602: Perform graph embedding on the gated feature map to obtain the embedded graph vector; Step S603: Perform position embedding on the gated feature map to obtain the embedded position vector; Step S604: Obtain the sum of the embedding graph vector and the embedding position vector to obtain the embedding feature vector; Step S605: Detect subtitle-free frames from the embedded feature vector to obtain video frame identifiers.

[0060] Steps S601 to S605 of this embodiment involve dividing the screened feature vectors into feature maps to obtain gated feature maps; embedding the gated feature maps into embedded graph vectors; embedding the gated feature maps into embedded position vectors; summing the embedded graph vectors and embedded position vectors to obtain embedded feature vectors; and performing subtitle-free frame detection on the embedded feature vectors to obtain video frame identifiers. Thus, this embodiment transforms the screened feature vectors into gated feature maps by dividing them into feature maps. Next, content information embedding and position information embedding are performed on the gated feature maps to obtain embedded graph vectors and embedded position vectors. These two vectors are summed to form the embedded feature vector, which integrates the content and position information of the video frame. Finally, the embedded feature vector is used to detect subtitle-free frames, thereby obtaining video frame identifiers and completing the detection of subtitle-free segments.

[0061] It should be noted that steps S601 to S605 are implemented by the TAD module using the Vision Transformer architecture.

[0062] In step S601 of some embodiments, the screened feature vector is first divided. For example, a 4×4 screened feature vector is divided into 2×2 image blocks, so a total of 4 gated feature maps can be obtained.

[0063] In step S602 of some embodiments, vector mapping is performed on each gated feature map using TAD to obtain the embedded graph vector.

[0064] In step S603 of some embodiments, position embedding is performed based on the position of the gated feature map in the sieved feature vector, such as sine embedding or cosine embedding, or position embedding is performed through a neural network such as MLP to obtain the embedded position vector.

[0065] In step S604 of some embodiments, for each gated feature map, its corresponding embedding map vector and embedding position vector are added together to obtain the embedding feature vector.

[0066] In step S605 of some embodiments, the detection of frames without subtitles is achieved by inputting the embedded feature vector into a Transformer encoder to calculate a category label, and then inputting the category label into a trained MLP to obtain a video frame identifier. For example, a video frame identifier can be 1, representing no subtitles, or 0, representing subtitles.

[0067] Please see Figure 7 In some embodiments, step S106 may include, but is not limited to, steps S701 to S703: Step S701: Calculate the edge weights based on the video frame identifier, change weights, and filtered feature vectors; whereby the edge weights are used to indicate the similarity between the original video frames. Step S702: Perform graph clustering on the original video frames according to the edge weights to obtain preliminary video segments without subtitles; Step S703: Perform non-maximum suppression processing on the initial video segment without subtitles to obtain the target video segment.

[0068] Steps S701 to S703, as illustrated in this embodiment, calculate edge weights reflecting the similarity between original video frames based on video frame identifiers, change weights, and filtered feature vectors. Then, the original video frames are clustered according to these edge weights, grouping similar frames into the same group, thus initially obtaining subtitle-free video segments. Further non-maximum suppression processing removes redundant segments, yielding the target video segment, thereby achieving accurate acquisition of subtitle-free video segments.

[0069] In step S701 of some embodiments, the edge weights are calculated based on the change weights and the sieved feature vectors, as shown in equation (5): (5), in, For subscript, For this is the first The and the first The edge weights between the sieved feature vectors This is a hyperparameter, with a default value of 0.1. The filtered feature vector Indicates the weight of the change. For video frame identification, Indicates the first The and the first All of them are videos without subtitles.

[0070] In step S702 of some embodiments, graph clustering is implemented using a temporal bipartite graph, in which each node is a video frame, and the weights between nodes are... In a temporal bipartite graph, nodes are clustered based on edge weights. Specifically, nodes with larger edge weights (i.e., nodes with higher similarity) are grouped together to find frames with similar characteristics in the graph. This method effectively groups similar frames into a single video segment, forming a preliminary subtitle-free video clip.

[0071] In step S703 of some embodiments, the initial subtitle-free video segments are clustered based on video frame similarity, i.e., edge weights. However, these segments may be redundant, especially some segments that may have significant temporal overlap or content repetition. By performing non-maximum suppression processing on the initial subtitle-free video segments, possible overlapping segments are eliminated, resulting in the target video segment.

[0072] Please see Figure 8 This application also provides a video segment extraction device that can implement the above-described video segment extraction method. The device includes: The data acquisition module 801 is used to acquire the original video and extract video frames from the original video to obtain the original video frames. The feature extraction module 802 is used to extract temporal features from the original video frames to obtain temporal feature vectors. The weight extraction module 803 is used to extract the change weights from the time series feature vector to obtain the change weights. The change filtering module 804 is used to filter the temporal feature vectors based on the change weights to obtain the filtered feature vectors. The frame detection module 805 is used to perform subtitle-free frame detection on the filtered feature vector to obtain video frame identifiers; the video frame identifiers are used to indicate whether the original video frame contains subtitles or not. The segment clustering module 806 is used to cluster the original video frames into segments based on the video frame identifier, change weight, and filtered feature vector to obtain the target video segments; wherein, the target video segments do not contain subtitles.

[0073] The specific implementation of this video clip extraction device is basically the same as the specific embodiment of the video clip extraction method described above, and will not be repeated here.

[0074] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described video segment extraction method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0075] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the video segment extraction method of the embodiments of this application. The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0076] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video segment extraction method.

[0077] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0078] The video segment extraction method, device, electronic device, and storage medium provided in this application embodiment acquire an original video and extract video frames from it to obtain original video frames. Next, temporal features are extracted from the original video frames to obtain temporal feature vectors. Subsequently, change weights are extracted from the temporal feature vectors to obtain change weights, and the temporal feature vectors are then filtered based on the change weights to obtain filtered feature vectors. Next, the filtered feature vectors are used to detect frames without subtitles to obtain video frame identifiers. Finally, the original video frames are clustered based on the video frame identifiers, change weights, and filtered feature vectors to obtain target video segments. Thus, this application embodiment, through temporal feature extraction and change weight extraction techniques, overcomes the limitations of existing technologies that rely on static frame analysis. Moreover, in dynamic scenes, it can better cope with the impact of factors such as changes in lighting and rapid transitions on detection accuracy, improving the detection accuracy of subtitle-free segments.

[0079] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0080] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0083] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0084] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0086] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for extracting video segments, characterized in that, The method includes: The original video is acquired, and video frames are extracted from the original video to obtain the original video frames; Temporal feature extraction is performed on the original video frames to obtain temporal feature vectors; The change weights are extracted from the time-series feature vectors to obtain the change weights. Based on the change weight, the temporal feature vector is filtered for image change to obtain the filtered feature vector; The filtered feature vectors are subjected to subtitle-free frame detection to obtain video frame identifiers; the video frame identifiers are used to indicate whether the original video frame contains or does not contain subtitles. The original video frames are clustered into segments based on the video frame identifier, the change weight, and the filtered feature vector to obtain target video segments; wherein, the target video segments do not contain subtitles.

2. The method according to claim 1, characterized in that, The step of extracting temporal features from the original video frames to obtain a temporal feature vector includes: Perform self-attention calculation on the original video frame to obtain at least one cross-window feature vector; Temporal fusion is performed on each of the cross-window feature vectors to obtain the temporal feature vector.

3. The method according to claim 2, characterized in that, The step of performing self-attention calculation on the original video frame to obtain at least one cross-window feature vector includes: The original video frames are divided into matrix blocks to obtain at least one three-dimensional window block; Self-attention calculation is performed on each of the three-dimensional window blocks to obtain the in-window self-attention feature vector; Calculate the similarity between the self-attention feature vectors within the window to obtain the window similarity; The cross-window feature vector is obtained by weighted fusion of the window similarity and the in-window self-attention feature vector, and then the cross-window feature vector is added to a preset vector set. The cross-window feature vector is used as the original video frame. The original video frame is then divided into matrix blocks to obtain at least one three-dimensional window block until the first preset iteration condition is met. Each vector in the vector set is used as the cross-window feature vector to obtain at least one cross-window feature vector.

4. The method according to claim 2, characterized in that, The cross-window feature vectors have sequential identifiers; the temporal fusion of each cross-window feature vector to obtain the temporal feature vector includes: Obtain the preset time-series state feature vector and the preset counter; Obtain the sequence identifier that is the same as the number in the counter to obtain the target identifier; The target cross-window vector is obtained from each of the cross-window feature vectors based on the target identifier; The target cross-window vector and the temporal state feature vector are fused in a temporal sequence to obtain a temporal fusion feature vector. The time-series fusion feature vector is updated according to the time-series state feature vector, the counter number is incremented according to the preset increment number, and the process of obtaining the sequence identifier that is the same as the counter number is returned to obtain the target identifier until the second preset iteration condition is reached. The time-series state feature vector is used as the time-series feature vector.

5. The method according to claim 1, characterized in that, The step of performing subtitle-free frame detection on the screened feature vectors to obtain video frame identifiers includes: The screened feature vectors are divided into feature maps to obtain gated feature maps; The gated feature map is embedded to obtain an embedded graph vector; The gated feature map is embedded to obtain an embedded position vector; The embedding feature vector is obtained by summing the embedding graph vector and the embedding position vector. The embedded feature vector is subjected to subtitle-free frame detection to obtain the video frame identifier.

6. The method according to claim 1, characterized in that, The step of clustering the original video frames according to the video frame identifier, the change weight, and the filtered feature vector to obtain the target video segments includes: The edge weights are calculated based on the video frame identifier, the change weights, and the filtered feature vectors; wherein the edge weights are used to indicate the similarity between the original video frames. Based on the edge weights, graph clustering is performed on the original video frames to obtain preliminary subtitle-free video segments; The initial video segment without subtitles is subjected to non-maximum suppression processing to obtain the target video segment.

7. The method according to any one of claims 1 to 6, characterized in that, The step of extracting video frames from the original video to obtain the original video frames includes: The original video is decoded to obtain decoded video frames; The decoded video frames are scaled to obtain scaled video frames; The scaled video frame is pixel-normalized to obtain the original video frame.

8. A video clip extraction device, characterized in that, The device includes: The data acquisition module is used to acquire the original video and extract video frames from the original video to obtain the original video frames. The feature extraction module is used to extract temporal features from the original video frames to obtain temporal feature vectors; The weight extraction module is used to extract the change weights from the time-series feature vector to obtain the change weights. The change amount filtering module is used to filter the temporal feature vector based on the change amount weight to obtain the filtered feature vector; The frame detection module is used to perform subtitle-free frame detection on the filtered feature vector to obtain a video frame identifier; the video frame identifier is used to indicate whether the original video frame contains subtitles or not. The segment clustering module is used to perform segment clustering on the original video frame based on the video frame identifier, the change weight, and the filtered feature vector to obtain target video segments; wherein, the target video segments do not contain subtitles.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the video segment extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video segment extraction method according to any one of claims 1 to 7.

Images