Video clip extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video clip extraction method and device, electronic equipment and a storage medium, belongs to the technical field of video processing, and is suitable for the fields of finance and medical treatment. The method comprises the following steps: acquiring an original video, and performing video frame extraction on the original video to obtain an original video frame; performing time sequence feature extraction on the original video frame to obtain a time sequence feature vector; performing variable quantity weight extraction on the time sequence feature vector to obtain a variable quantity weight; performing picture variable quantity screening on the time sequence feature vector according to the variable quantity weight to obtain a screened feature vector; performing non-subtitle frame detection on the screened feature vector to obtain a video frame identifier; and performing fragment clustering on the original video frame according to the video frame identifier, the variable quantity weight and the post-screening feature vector to obtain a target video fragment. According to the embodiment of the invention, the subtitle-free video clip in the video can be accurately detected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and is suitable for the fields of finance and medicine, and in particular relates to a video segment extraction method and device, an electronic device and a storage medium. BACKGROUND

[0002] A subtitle-free video segment refers to a segment in a video that does not have subtitles. Identifying a subtitle-free segment can help push more accurate advertising content. For example, in an insurance product advertisement, identifying a subtitle-free segment can accurately insert an advertisement without reducing the visual perception of the video. For example, in a medical teaching scenario, identifying a subtitle-free segment can accurately insert a medical device name. In existing subtitle detection methods, one method is to identify the text in a video frame through OCR, but due to the text on a street sign or a sign in the video frame, the recognition accuracy is low. Another method is to identify a subtitle-free segment through a neural network, but most methods rely on static frame analysis, and due to the influence of dynamic scenes such as light and shadow changes and fast transitions in the video, the static frame analysis effect is poor. Therefore, how to accurately detect a subtitle-free video segment in a video has become a problem to be solved. SUMMARY

[0003] The main purpose of the embodiments of the present application is to propose a video segment extraction method and device, an electronic device and a storage medium, which aims to accurately detect a subtitle-free video segment in a video.

[0004] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application proposes a video segment extraction method, which comprises: obtaining an original video and performing video frame extraction on the original video to obtain original video frames; performing time sequence feature extraction on the original video frames to obtain time sequence feature vectors; performing change amount weight extraction on the time sequence feature vectors to obtain change amount weights; performing picture change amount screening on the time sequence feature vectors according to the change amount weights to obtain screened feature vectors; performing subtitle-free frame detection on the screened feature vectors to obtain video frame identifiers; the video frame identifiers are used to indicate whether the original video frames contain subtitles or not; performing segment clustering on the original video frames according to the video frame identifiers, the change amount weights and the screened feature vectors to obtain target video segments; wherein the target video segments do not contain subtitles.

[0005] In some embodiments, the time sequence feature extraction on the original video frames to obtain time sequence feature vectors comprises: performing self-attention calculation on the original video frames to obtain at least one cross-window feature vector; performing time sequence fusion on each of the cross-window feature vectors to obtain the time sequence feature vector.

[0006] In some embodiments, the performing self-attention calculation on the original video frame to obtain at least one cross-window feature vector comprises: performing matrix blocking on the original video frame to obtain at least one three-dimensional window block; performing self-attention calculation on each of the three-dimensional window blocks to obtain intra-window self-attention feature vectors; calculating similarity between the intra-window self-attention feature vectors to obtain window similarity; performing weighted fusion according to the window similarity and the intra-window self-attention feature vectors to obtain the cross-window feature vector, and adding the cross-window feature vector to a preset vector set; taking the cross-window feature vector as the original video frame to return to perform the matrix blocking on the original video frame to obtain at least one three-dimensional window block until a first preset iteration condition is reached; taking each vector in the vector set as the cross-window feature vector to obtain at least one cross-window feature vector.

[0007] In some embodiments, the cross-window feature vector has a sequential identifier; and the performing time sequence fusion on each of the cross-window feature vectors to obtain the time sequence feature vector comprises: obtaining a preset time sequence state feature vector and a preset counter; obtaining a sequential identifier same as a number of the counter to obtain a target identifier; obtaining a target cross-window vector from each of the cross-window feature vectors according to the target identifier; performing time sequence fusion on the time sequence state feature vector according to the target cross-window vector to obtain a time sequence fusion feature vector; updating the time sequence state feature vector according to the time sequence fusion feature vector, and performing self-increment calculation on the number of the counter according to a preset self-increment number, and returning to perform the obtaining a sequential identifier same as the number of the counter to obtain a target identifier until a second preset iteration condition is reached; taking the time sequence state feature vector as the time sequence feature vector.

[0008] In some embodiments, the performing subtitle-free frame detection on the screened feature vector to obtain a video frame identifier comprises: performing feature map division on the screened feature vector to obtain a gating feature map; performing graph embedding on the gating feature map to obtain an embedded graph vector; perform position embedding on the gated feature map to obtain an embedded position vector; obtain a sum of the embedded graph vector and the embedded position vector to obtain an embedded feature vector; perform subtitle-free frame detection on the embedded feature vector to obtain the video frame identifier.

[0009] In some embodiments, the segment clustering of the original video frame according to the video frame identifier, the change amount weight, and the screened feature vector obtains a target video segment, including: perform calculation according to the video frame identifier, the change amount weight, and the screened feature vector to obtain an edge weight; wherein the edge weight is used to indicate the similarity between the original video frames; perform graph clustering on the original video frame according to the edge weight to obtain a preliminary subtitle-free video segment; perform non-maximum suppression processing on the preliminary subtitle-free video segment to obtain the target video segment.

[0010] In some embodiments, the video frame extraction of the original video obtains an original video frame, including: perform data decoding on the original video to obtain a decoded video frame; perform size scaling on the decoded video frame to obtain a scaled video frame; perform pixel normalization on the scaled video frame to obtain the original video frame.

[0011] To achieve the above object, a second aspect of the embodiment of the present application proposes a video segment extraction device, the device comprising: an acquisition data module configured to acquire an original video and perform video frame extraction on the original video to obtain an original video frame; a feature extraction module configured to perform time sequence feature extraction on the original video frame to obtain a time sequence feature vector; a weight extraction module configured to perform change amount weight extraction on the time sequence feature vector to obtain a change amount weight; a change amount screening module configured to perform picture change amount screening on the time sequence feature vector according to the change amount weight to obtain a screened feature vector; a frame detection module configured to perform subtitle-free frame detection on the screened feature vector to obtain a video frame identifier; the video frame identifier is used to indicate whether the original video frame contains a subtitle or not; a segment clustering module configured to perform segment clustering on the original video frame according to the video frame identifier, the change amount weight, and the screened feature vector to obtain a target video segment; wherein the target video segment does not contain a subtitle.

[0012] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0013] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0014] The video segment extraction method and device, electronic device and storage medium provided by the present application can obtain the original video and extract the video frames of the original video, then extract the time sequence features of the original video frames to obtain the time sequence feature vectors, then extract the change weight of the time sequence feature vectors to obtain the change weight, and screen the time sequence feature vectors according to the change weight to obtain the screened feature vectors, then detect the subtitle-free frames by using the screened feature vectors to obtain the video frame identifier, and finally cluster the original video frames according to the video frame identifier, the change weight and the screened feature vectors to obtain the target video segment. Therefore, the embodiments of the present application break through the limitation of the prior art which depends on the static frame analysis, and can better cope with the influence of light and shadow changes, fast scene transitions and other factors on the detection accuracy in the dynamic scene, thereby improving the detection accuracy of the subtitle-free segment. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of the video segment extraction method provided by the embodiments of the present application; Figure 2 is a flowchart of step S101 in Figure 1 ; Figure 3 is a flowchart of step S102 in Figure 1 ; Figure 4 is a flowchart of step S301 in Figure 3 ; Figure 5 is a flowchart of step S302 in Figure 3 ; Figure 6 is a flowchart of step S105 in Figure 1 ; Figure 7 is a flowchart of step S106 in Figure 1 ; Figure 8 is a structural schematic diagram of the video segment extraction device provided by the embodiments of the present application; Figure 9 FIG. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application and not to limit the present application.

[0017] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0019] The subtitle-free video segment refers to a segment in a video without subtitles. Identifying the subtitle-free segment can help to push more accurate advertising content. For example, in an insurance product advertisement, identifying the subtitle-free segment can accurately insert the advertisement without reducing the video visual effect. In existing subtitle detection methods, one method is to identify the text in the video frame through OCR, but due to the text on the signboard or the signboard in the video frame, the recognition accuracy is low. Another method is to identify the subtitle-free segment through a neural network, but most methods rely on static frame analysis, and due to the influence of dynamic scenes such as light and shadow changes and fast transitions in the video, the static frame analysis effect is poor. Therefore, how to accurately detect the subtitle-free video segment in the video has become a problem to be solved.

[0020] Based on this, the embodiments of the present application provide a video segment extraction method and device, an electronic device and a storage medium, which are aimed at accurately detecting the subtitle-free video segment in the video.

[0021] The video segment extraction method and device, the electronic device and the storage medium provided by the embodiments of the present application are specifically described through the following embodiments. First, the video segment extraction method in the embodiments of the present application is described.

[0022] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving environment, acquiring knowledge and using knowledge to obtain optimal results.

[0023] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0024] The video segment extraction method provided by the embodiments of the present application relates to the technical field of video processing and is suitable for the financial and medical fields. The video segment extraction method provided by the embodiments of the present application can be applied in a terminal, can be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN and big data and artificial intelligence platform; and the software can be an application for implementing the video segment extraction method, etc., but is not limited to the above forms.

[0025] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0026] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or separate consent will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0027] Figure 1 is an optional flowchart of the video segment extraction method provided by the embodiments of the present application, Figure 1 The method in the above step S100 can include but is not limited to steps S101-S106.

[0028] Step S101, obtaining an original video and performing video frame extraction on the original video to obtain original video frames; Step S102, performing time sequence feature extraction on the original video frames to obtain time sequence feature vectors; Step S103, performing change weight extraction on the time sequence feature vectors to obtain change weights; Step S104, performing picture change amount screening on the time sequence feature vectors according to the change weights to obtain screened feature vectors; Step S105, performing subtitle-free frame detection on the screened feature vectors to obtain video frame identifiers; the video frame identifiers are used to indicate whether the original video frames contain subtitles or not; Step S106, performing segment clustering on the original video frames according to the video frame identifiers, the change weights, and the screened feature vectors to obtain target video segments; wherein the target video segments do not contain subtitles.

[0029] The steps S101 to S106 shown in the embodiments of the present application are as follows: the original video is obtained and video frame extraction is performed on the original video to obtain original video frames. Then, time sequence feature extraction is performed on the original video frames to obtain time sequence feature vectors. Subsequently, the change weight is obtained by performing change weight extraction on the time sequence feature vectors, and the filtered feature vectors are obtained by performing picture change amount filtering on the time sequence feature vectors according to the change weight. Next, the video frame identifier is obtained by using the filtered feature vectors to perform subtitle-free frame detection. Finally, the original video frames are clustered according to the video frame identifier, the change weight and the filtered feature vectors, so as to obtain the target video segment. In this way, the embodiments of the present application break through the limitation of relying on static frame analysis in the prior art, and can better cope with the influence of factors such as light and shadow changes and fast transitions on detection accuracy in a dynamic scene, thereby improving the detection accuracy of the subtitle-free segment.

[0030] Please refer to Figure 2 In some embodiments, step S101 can include but is not limited to steps S201 to S204: Step S201, data decoding is performed on the original video to obtain decoded video frames; Step S202, size scaling is performed on the decoded video frames to obtain scaled video frames; Step S203, pixel normalization is performed on the scaled video frames to obtain original video frames.

[0031] The steps S201 to S204 shown in the embodiments of the present application are as follows: data decoding is performed on the original video to obtain decoded video frames; then, size scaling is performed on the decoded video frames to obtain scaled video frames; and then, pixel normalization is performed on the scaled video frames to obtain original video frames. In this way, the embodiments of the present application extract image frames from the video through decoding operation, and then perform size scaling and pixel normalization on the image frames to unify the size and pixel value of the input data, so as to ensure the data format consistency of the input model and avoid the model performance fluctuation caused by size and pixel difference.

[0032] In step S201 of some embodiments, data decoding refers to reading the original video file by using the OpenCV video processing library and loading it into the memory to obtain the decoded video frames. The decoded image frames are usually image data in RGB or YUV format.

[0033] In step S202 of some embodiments, size scaling processing refers to scaling the decoded video frames to a size of 224x224 to obtain scaled video frames.

[0034] In step S203 of some embodiments, performing pixel normalization processing refers to normalizing image pixel values into an interval of [0, 1], thereby eliminating the brightness difference between different video frames and avoiding algorithm performance fluctuations caused by different image pixel values, thereby improving the detection accuracy of the subtitle-free video frame.

[0035] Referring to Figure 3 In some embodiments, step S102 can include but is not limited to steps S301 to S302: Step S301, performing self-attention calculation on the original video frame to obtain at least one cross-window feature vector; Step S302, performing time sequence fusion on each cross-window feature vector to obtain a time sequence feature vector.

[0036] The steps S301 to S302 shown in the embodiments of the present application can extract relevant features between different windows by performing self-attention calculation on the original video frame, obtain at least one cross-window feature vector, capture long-distance dependency between different frames, and improve the accuracy and richness of feature extraction. Then, the information in the time sequence is integrated by performing time sequence fusion on each cross-window feature vector to form a time sequence feature vector, ensuring that the time dependence of the video can be maintained when processing the video frame, thereby better reflecting the dynamic changes in the video.

[0037] Referring to Figure 4 In some embodiments, step S301 can include but is not limited to steps S401 to S406: Step S401, performing matrix blocking on the original video frame to obtain at least one three-dimensional window block; Step S402, performing self-attention calculation on each three-dimensional window block to obtain an intra-window self-attention feature vector; Step S403, calculating the similarity between the intra-window self-attention feature vectors to obtain a window similarity; Step S404, performing weighted fusion according to the window similarity and the intra-window self-attention feature vector to obtain a cross-window feature vector, and adding the cross-window feature vector to a preset vector set; Step S405, taking the cross-window feature vector as the original video frame, returning to perform matrix blocking on the original video frame to obtain at least one three-dimensional window block, until a first preset iteration condition is reached; Step S406, taking each vector in the vector set as a cross-window feature vector to obtain at least one cross-window feature vector.

[0038] The steps S401 to S406 shown in the embodiments of the present application divide the video frame into a plurality of three-dimensional window blocks by matrix blocking on the original video frame, independently calculate the intra-window self-attention feature vector of each three-dimensional window block through the self-attention mechanism, and extract the spatial and temporal features in each window block. After calculating the self-attention feature in each three-dimensional window block, the similarity between the intra-window self-attention feature vectors is calculated to obtain the window similarity, and then the window similarity and the self-attention feature vector are weighted and fused to obtain the cross-window feature vector. Finally, the self-attention calculation and weighted fusion of the three-dimensional window block in the embodiments of the present application enable each window block to be processed independently while retaining the local features and strengthening the correlation with other window blocks.

[0039] It should be noted that the steps S401 to S406 are processed by the 3D Swin Transformer backbone network, and the 3D Swin Transformer includes a plurality of Swin3Dblocks, and each Swin3Dblock is connected in cascade.

[0040] In the steps S401 to S406 of some embodiments, for the first Swin3Dblock in the 3D Swin Transformer backbone network, the original video frame is input to the Swin3Dblock for matrix blocking to obtain a plurality of three-dimensional window blocks. For example, assuming that the size of the original video frame is HxWxC (where H is the height, W is the width, and C is the number of channels), the original video frame is blocked according to the set window size (for example, 4x4x4) to obtain a plurality of 4x4x4 three-dimensional window blocks, and the three-dimensional window block is a matrix.

[0041] The self-attention weight of each three-dimensional window block is independently calculated by the Swin3Dblock to obtain the intra-window self-attention feature vector of each three-dimensional window block. This independent calculation method can enhance the information extraction capability of each window block in a specific local area. For example, different areas and time frames in a video segment may contain different types of motion or changes. Through self-attention calculation, each three-dimensional window block can be weighted according to the specific pattern inside it, thereby better capturing the local spatial and temporal information.

[0042] The similarity between the intra-window self-attention feature vectors is calculated to obtain the window similarity, as shown in equation (1): (1), wherein, is the intra-window self-attention feature vector, is the first window, is the second window, the similarity between windows.

[0043] The weighted fusion according to the window similarity and the intra-window self-attention feature vector is shown in formula (2): (2), wherein, is the cross-window feature vector. is the number of windows. For example, there are three three-dimensional window blocks, and the corresponding intra-window self-attention feature vectors are , the similarity between the first window and the second window is , the similarity between the second and the first is , and similarly, there are . Then, for the first window, the corresponding is . Then similarly, the cross-window feature vector of the second window and the cross-window feature vector of the third window can be obtained.

[0044] Then the first, second and third cross-window feature vectors are marked with the label of the layer, for example, the cross-window feature vectors of the first layer Swin3Dblock, and are added to the vector set.

[0045] Then the cross-window feature vectors are spliced back to the original position in the video frame according to the corresponding three-dimensional window block, replacing the original video frame, and input to the next layer of Swin3Dblock, and the process is repeated until the cross-window feature vector of each layer of Swin3Dblock is obtained, that is, the first preset iteration condition is reached, and finally all vectors in the vector set are taken as the cross-window feature vector.

[0046] On the other hand, the embodiments of the present application realize the step-by-step extraction from shallow features to deep features by cascading processing between multiple Swin3Dblocks. In each layer of Swin3Dblock, the original video frame is processed by matrix blocking, intra-window self-attention calculation and cross-window feature fusion to generate a cross-window feature vector, which is transmitted to the next layer of Swin3Dblock for further processing. Through this hierarchical structure, the model can gradually integrate more local information and enhance the expression ability of features layer by layer.

[0047] Referring to Figure 5 In some embodiments, the cross-window feature vector has a sequential identifier, and step S302 includes but is not limited to steps S501 to S506: Step S501, obtaining a preset time sequence state feature vector and a preset counter; Step S502, obtaining a sequential identifier same as the number of the counter to obtain a target identifier; Step S503, obtaining a target cross-window vector from each cross-window feature vector according to the target identifier; Step S504, performing time sequence fusion on the time sequence state feature vector according to the target cross-window vector to obtain a time sequence fusion feature vector; Step S505, updating the time sequence state feature vector according to the time sequence fusion feature vector, and performing self-increment calculation on the number of the counter according to a preset self-increment number, returning to execute the step of obtaining a sequential identifier same as the number of the counter to obtain a target identifier until a second preset iteration condition is reached; Step S506, taking the time sequence state feature vector as a time sequence feature vector.

[0048] The steps S501 to S506 shown in the embodiments of the present application obtain a preset time sequence state feature vector and a preset counter, obtain a sequential identifier same as the number of the counter to obtain a target identifier, obtain a target cross-window vector from each cross-window feature vector according to the target identifier, perform time sequence fusion on the time sequence state feature vector according to the target cross-window vector to obtain a time sequence fusion feature vector, update the time sequence state feature vector according to the time sequence fusion feature vector, perform self-increment calculation on the number of the counter according to a preset self-increment number, return to execute the step of obtaining a sequential identifier same as the number of the counter to obtain a target identifier until a second preset iteration condition is reached, and take the time sequence state feature vector as a time sequence feature vector. In this way, the embodiments of the present application fuse each cross-window feature vector in the form of time sequence information by taking the sequence between windows as time sequence information, to obtain a time sequence feature vector representing a video frame.

[0049] It should be noted that the steps S501 to S506 are performed by the LSTM network for time sequence fusion.

[0050] In the steps S501 to S506 of some embodiments, first, a preset time sequence state feature vector is obtained, which can be initialized. The counter starts from 1 (in a computer, it starts from 0 in binary, which is used to represent the first layer Swin3Dblock). Each Swin3Dblock has a sequential identifier, the Swin3Dblock corresponding to the counter is obtained, and then the target cross-window vector corresponding to the Swin3Dblock is obtained, for example, the first layer Swin3Dblock has three target cross-window vectors .

[0051] The time sequence state feature vector is fused according to the target cross-window vector to obtain a time sequence fusion feature vector, for example, the time sequence state feature vector and Input them together into the LSTM to obtain , and Input into LSTM, get ,Will and Input into LSTM, get ,but This is the time-series feature vector corresponding to a counter of 1.

[0052] Then, the speedometer is incremented by 1, and the process of calculating the temporal feature vector of the second-layer Swin3Dblock is returned. This is the temporal state feature vector. When the counter is 2 Input them together into the LSTM to obtain , When the counter is 2 Input into LSTM, get ,Will When the counter is 2 Input into LSTM, get ,but This is the time-series feature vector corresponding to a counter of 2.

[0053] It should be noted that in each layer, the temporal state feature vector They are all the same as when obtaining the preset temporal state feature vector in step S501, and do not change with the traversal of each layer of Swin3Dblock.

[0054] The process continues until all Swing3Dblocks have been traversed, reaching the second preset iteration condition.

[0055] Finally, all layers corresponding to As temporal feature vectors, a vector set is obtained, the length of which is the same as that of the Swin3Dblock, denoted as . ,in is the length of the vector set.

[0056] In step S103 of some embodiments, the change weight extraction is generated by motion consistency analysis and implemented by the motion conditional attention gating module (MCAM). The specific calculation in the motion conditional attention gating module is shown in equation (3): (3), in, Indicates the first The weights of the changes in each time series feature vector. This is the preset sliding window size, which defaults to 8. is a sigmoid activation function. is a learnable weight network of the motion-condition attention gate module. represents the i-th temporal feature vector. represents the i-th temporal feature vector. represents the L2 norm measure of temporal continuity, and adaptively suppresses areas that do not meet the motion consistency (such as flash transitions), and realizes analysis from static frames to dynamic frames.

[0057] In step S104 of some embodiments, the picture change amount screening is as shown in formula (4): (4), wherein, is the screened feature vector. represents the i-th temporal feature vector. represents the i-th temporal feature vector.

[0058] Through steps S103 to S104, through the motion consistency analysis and the motion-condition attention gate module, the motion change information in the video is accurately extracted, and the irrelevant dynamic changes (such as transitions or light changes, etc.) are effectively suppressed through the screening of the change amount weight. Thus, the detection accuracy of subsequent subtitle-free video detection is improved.

[0059] Please refer to Figure 6 In some embodiments, step S105 includes but is not limited to steps S601 to S605: Step S601, dividing the screened feature vector into feature maps to obtain a gating feature map; Step S602, performing graph embedding on the gating feature map to obtain an embedded graph vector; Step S603, performing position embedding on the gating feature map to obtain an embedded position vector; Step S604, obtaining the sum of the embedded graph vector and the embedded position vector to obtain an embedded feature vector; Step S605, performing subtitle-free frame detection on the embedded feature vector to obtain a video frame identifier.

[0060] The steps S601 to S605 shown in the embodiments of the present application perform feature map division on the screened feature vectors to obtain gated feature maps; perform graph embedding on the gated feature maps to obtain embedded graph vectors; perform position embedding on the gated feature maps to obtain embedded position vectors; obtain the sum of the embedded graph vectors and the embedded position vectors to obtain an embedded feature vector; and perform subtitle-free frame detection on the embedded feature vector to obtain a video frame identifier. In this way, the embodiments of the present application convert the screened feature vectors into gated feature maps by performing feature map division on the screened feature vectors. Then, content information embedding and position information embedding are respectively performed on the gated feature maps to obtain embedded graph vectors and embedded position vectors. The two vectors are summed to form an embedded feature vector, which integrates the content information and the position information of the video frames. Finally, subtitle-free frame detection is performed using the embedded feature vector to obtain a video frame identifier, thereby completing the detection of the subtitle-free segment.

[0061] It should be noted that in steps S601 to S605, the TAD module adopting the Vision Transformer architecture is used.

[0062] In step S601 of some embodiments, the screened feature vectors are first divided, for example, a 4x4 screened feature vector is divided into 2x2 image blocks, and a total of 4 gated feature maps can be obtained.

[0063] In step S602 of some embodiments, the TAD is used to map the vectors of each gated feature map to obtain an embedded graph vector.

[0064] In step S603 of some embodiments, according to the position of the gated feature map in the screened feature vector, position embedding is performed, for example, sine embedding or cosine embedding, or position embedding is performed through a neural network such as an MLP, to obtain an embedded position vector.

[0065] In step S604 of some embodiments, for each gated feature map, the corresponding embedded graph vector and embedded position vector are added to obtain an embedded feature vector.

[0066] In step S605 of some embodiments, subtitle-free frame detection is performed by inputting the embedded feature vector into a Transformer encoder to calculate a class label, and then inputting the class label into a trained MLP to obtain a video frame identifier. For example, the video frame identifier is 1, representing no subtitle, or 0, representing subtitle.

[0067] Please refer to Figure 7 In some embodiments, step S106 can include but is not limited to steps S701 to S703: Step S701, calculating according to the video frame identifier, the change weight and the screened feature vector to obtain an edge weight; wherein the edge weight is used to indicate the similarity between the original video frames; Step S702, performing graph clustering on the original video frames according to the edge weight to obtain a preliminary subtitle-free video segment; Step S703, performing non-maximum suppression processing on the preliminary subtitle-free video segment to obtain a target video segment.

[0068] The steps S701 to S703 shown in the embodiments of the present application, by calculating according to the video frame identifier, the change weight and the screened feature vector, the edge weight reflecting the similarity between the original video frames is obtained. Then, according to the edge weight, the original video frames are clustered, which can divide similar frames into the same group, thereby obtaining the preliminary subtitle-free video segment. Through further non-maximum suppression processing, the redundant segments are removed to obtain the target video segment, thereby realizing the accurate acquisition of the subtitle-free video segment.

[0069] In step S701 of some embodiments, the edge weight is calculated according to the change weight and the screened feature vector, as shown in formula (5): (5), wherein, is a subscript, is the edge weight between the th and the th screened feature vector, is a hyperparameter, and the default is 0.1, is the screened feature vector, indicates the change weight, is the video frame identifier, indicates that the th and the th are both subtitle-free videos.

[0070] In step S702 of some embodiments, the graph clustering is realized by a time sequence bipartite graph. In the time sequence bipartite graph, each node is a video frame, and the weight between the nodes is In the time sequence bipartite graph, the nodes are clustered according to the edge weight. The specific steps are as follows: the nodes with larger edge weight (i.e. the nodes with higher similarity) are gathered together, so that a group of frames with similar features in the graph is found. Through this method, similar frames can be effectively gathered into a video segment to form a preliminary subtitle-free video segment.

[0071] In step S703 of some embodiments, the preliminary subtitle-free video segments are clustered based on the video frame similarity, i.e., the edge weight. However, these segments can have redundancy, especially some segments can have large overlap in time or be repetitive in content. By performing non-maximum suppression on the preliminary subtitle-free video segments, the possible overlapping segments are eliminated, and the target video segments are obtained.

[0072] Please refer to Figure 8 The embodiments of the present application also provide a video segment extraction device, which can implement the above video segment extraction method. The device comprises: The data acquisition module 801 is configured to acquire an original video and perform video frame extraction on the original video to obtain original video frames. The feature extraction module 802 is configured to perform time sequence feature extraction on the original video frames to obtain time sequence feature vectors. The weight extraction module 803 is configured to perform change weight extraction on the time sequence feature vectors to obtain change weights. The change screening module 804 is configured to perform picture change screening on the time sequence feature vectors according to the change weights to obtain screened feature vectors. The frame detection module 805 is configured to perform subtitle-free frame detection on the screened feature vectors to obtain video frame identifiers. The video frame identifiers are used to indicate whether the original video frames contain subtitles or not. The segment clustering module 806 is configured to perform segment clustering on the original video frames according to the video frame identifiers, the change weights, and the screened feature vectors to obtain target video segments. The target video segments do not contain subtitles.

[0073] The specific implementation of the video segment extraction device is basically the same as the specific embodiments of the above video segment extraction method, and will not be repeated here.

[0074] The embodiments of the present application also provide an electronic device. The electronic device comprises a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the above video segment extraction method is implemented. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0075] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated. The electronic device comprises: The processor 901 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application. The memory 902 can be implemented by a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and the like. The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the video segment extraction method of the embodiments of the present application. The input / output interface 903 is configured to realize information input and output. The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, and the like). The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device. The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0076] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned video segment extraction method.

[0077] The memory is a non-transient computer readable storage medium, which can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transient memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transient solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0078] The video segment extraction method, the video segment extraction device, the electronic equipment and the storage medium provided by the embodiment of the application are provided. The original video is acquired, and video frame extraction is performed on the original video to obtain original video frames. Then, time sequence feature extraction is performed on the original video frames to obtain time sequence feature vectors. Subsequently, change weight extraction is performed on the time sequence feature vectors to obtain change weights, and picture change filtering is performed on the time sequence feature vectors according to the change weights to obtain filtered feature vectors. Next, the filtered feature vectors are used for subtitle-free frame detection to obtain video frame identifiers. Finally, segment clustering is performed on the original video frames according to the video frame identifiers, the change weights and the filtered feature vectors, so that target video segments are obtained. In this way, the embodiment of the application breaks through the limitation of the prior art that depends on static frame analysis by using time sequence feature extraction and change weight extraction technologies, and can better cope with the influence of factors such as light and shadow changes and fast scene transitions on detection accuracy in a dynamic scene, thereby improving the detection accuracy of subtitle-free segments.

[0079] The embodiments described in the embodiments of the application are used to more clearly illustrate the technical solutions of the embodiments of the application, and do not constitute a limitation on the technical solutions provided by the embodiments of the application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0080] Those skilled in the art can understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the application, and can include more or fewer steps than the drawings, or combine certain steps or different steps.

[0081] The device embodiments described above are only schematic, and the units illustrated as separate components can or can not be physically separated, that is, can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the application.

[0082] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0083] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but can be used for clarity, and merely establishes the order unless otherwise stated below.

[0084] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0085] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0086] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0087] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0088] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0089] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A video segment extraction method characterized by, The method comprises: acquiring an original video and performing video frame extraction on the original video to obtain original video frames; performing time sequence feature extraction on the original video frames to obtain time sequence feature vectors; performing change weight extraction on the time sequence feature vectors to obtain change weights; performing picture change filtering on the time sequence feature vectors according to the change weights to obtain filtered feature vectors; performing subtitle-free frame detection on the filtered feature vectors to obtain video frame identifiers; the video frame identifiers are used to indicate whether the original video frames contain subtitles or not; performing segment clustering on the original video frames according to the video frame identifiers, the change weights and the filtered feature vectors to obtain target video segments; wherein the target video segments do not contain subtitles.

2. The method of claim 1, wherein, The time sequence feature extraction on the original video frames to obtain time sequence feature vectors comprises: performing self-attention calculation on the original video frames to obtain at least one cross-window feature vector; performing time sequence fusion on each cross-window feature vector to obtain the time sequence feature vectors.

3. The method of claim 2, wherein, The self-attention calculation on the original video frames to obtain at least one cross-window feature vector comprises: performing matrix block division on the original video frames to obtain at least one three-dimensional window block; performing self-attention calculation on each three-dimensional window block to obtain intra-window self-attention feature vectors; calculating the similarity between the intra-window self-attention feature vectors to obtain window similarity; performing weighted fusion according to the window similarity and the intra-window self-attention feature vectors to obtain the cross-window feature vector, and adding the cross-window feature vector to a preset vector set; taking the cross-window feature vector as the original video frame to return to perform the matrix block division on the original video frames to obtain at least one three-dimensional window block until a first preset iteration condition is reached; taking each vector in the vector set as the cross-window feature vector to obtain at least one cross-window feature vector.

4. The method of claim 2, wherein, The cross-window feature vector has a sequential identifier; the time sequence fusion on each cross-window feature vector to obtain the time sequence feature vectors comprises: acquiring a preset time sequence state feature vector and a preset counter; acquiring a sequential identifier identical to the number of the counter to obtain a target identifier; acquiring a target cross-window vector from each cross-window feature vector according to the target identifier; performing time sequence fusion on the time sequence state feature vector according to the target cross-window vector to obtain a time sequence fusion feature vector; updating the time sequence state feature vector according to the time sequence fusion feature vector, and performing self-increment calculation on the number of the counter according to a preset self-increment number to return to perform the acquisition of the sequential identifier identical to the number of the counter to obtain the target identifier until a second preset iteration condition is reached; taking the time sequence state feature vector as the time sequence feature vector.

5. The method of claim 1, wherein, The subtitle-free frame detection on the filtered feature vectors to obtain video frame identifiers comprises: performing feature map division on the filtered feature vectors to obtain gating feature maps; perform graph embedding on the gated feature map to obtain an embedding graph vector; perform position embedding on the gated feature map to obtain an embedding position vector; obtain a sum of the embedding graph vector and the embedding position vector to obtain an embedding feature vector; perform subtitle-free frame detection on the embedding feature vector to obtain the video frame identifier.

6. The method of claim 1, wherein, The segment clustering of the original video frame according to the video frame identifier, the change amount weight, and the screened feature vector includes: performing calculation according to the video frame identifier, the change amount weight, and the screened feature vector to obtain an edge weight; wherein the edge weight is used to indicate the similarity between the original video frames; performing graph clustering on the original video frames according to the edge weight to obtain a preliminary subtitle-free video segment; performing non-maximum suppression processing on the preliminary subtitle-free video segment to obtain the target video segment.

7. The method according to any one of claims 1 to 6, characterized in that, The original video frame extraction of the original video includes: performing data decoding on the original video to obtain a decoded video frame; performing size scaling on the decoded video frame to obtain a scaled video frame; performing pixel normalization on the scaled video frame to obtain the original video frame.

8. A video segment extraction apparatus characterized by comprising: The device includes: an acquisition data module configured to acquire an original video and perform original video frame extraction on the original video to obtain an original video frame; a feature extraction module configured to perform time sequence feature extraction on the original video frame to obtain a time sequence feature vector; a weight extraction module configured to perform change amount weight extraction on the time sequence feature vector to obtain a change amount weight; a change amount screening module configured to perform picture change amount screening on the time sequence feature vector according to the change amount weight to obtain a screened feature vector; a frame detection module configured to perform subtitle-free frame detection on the screened feature vector to obtain a video frame identifier; wherein the video frame identifier is used to indicate whether the original video frame contains a subtitle or not; a segment clustering module configured to perform segment clustering on the original video frame according to the video frame identifier, the change amount weight, and the screened feature vector to obtain a target video segment; wherein the target video segment does not contain a subtitle.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the video segment extraction method in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the video segment extraction method in any one of claims 1 to 7.

Images