Key frame detection method, device, equipment, storage medium, and program product

Abstract

The present application provides a key frame detection method, apparatus, device, storage medium, and program product; embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and vehicle-mounted, and involve artificial intelligence technology; the method includes: extracting features from the first video frame and the second video frame of the video to be detected, respectively, to obtain first video frame features of the first video frame and second video frame features of the second video frame; constructing temporal features and spatial features based on the first video frame features and the second video frame features; determining the information similarity between the first video frame and the second video frame using the temporal features and the spatial features; and determining the key frame corresponding to the local information in the video to be detected based on the information similarity. Through the present application, the accuracy of key frame detection can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a key frame detection method, device, equipment, storage medium, and program product. Background Art

[0002] Key frame detection refers to the process of selecting a small number of representative video frames or video segments in a video. It can remove redundancy in the video. Therefore, it can be widely used in scenarios such as video understanding and video infringement analysis.

[0003] In some use cases, many video processing techniques rely on local information within a video, such as text, to detect keyframes. However, in related technologies, determining keyframes based on local information suffers from low precision in calculating the similarity of local information, resulting in low keyframe detection accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a key frame detection method, apparatus, device, computer-readable storage medium, and program product, which can improve the accuracy of key frame detection.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a key frame detection method, including:

[0007] Performing feature extraction on a first video frame and a second video frame of the video to be detected, respectively, to obtain a first video frame feature of the first video frame and a second video frame feature of the second video frame;

[0008] Based on the first video frame features and the second video frame features, constructing a temporal feature and a spatial feature; wherein the temporal feature represents a position change of local information in the video to be detected between the first video frame and the second video frame, and the spatial feature represents an appearance of the local information in the first video frame and the second video frame;

[0009] Determining information similarity between the first video frame and the second video frame using the temporal features and the spatial features;

[0010] According to the information similarity, a key frame corresponding to the local information in the video to be detected is determined.

[0011] The present invention provides a key frame detection device, comprising:

[0012] a feature extraction module, configured to extract features from a first video frame and a second video frame of the video to be detected, respectively, to obtain first video frame features of the first video frame and second video frame features of the second video frame;

[0013] a feature construction module, configured to construct a temporal feature and a spatial feature based on the first video frame feature and the second video frame feature; wherein the temporal feature represents a position change of local information in the video to be detected between the first video frame and the second video frame, and the spatial feature represents an appearance of the local information in the first video frame and the second video frame;

[0014] a similarity determination module, configured to determine information similarity between the first video frame and the second video frame using the temporal features and the spatial features;

[0015] The key frame determination module is used to determine the key frame corresponding to the local information in the video to be detected based on the information similarity.

[0016] In some embodiments of the present application, the feature construction module is also used to determine a first mask area representing the position of the local information in the first video frame based on the first video frame feature, and to determine a second mask area representing the position of the local information in the second video frame based on the second video frame feature; to fuse the first mask area and the first video frame feature into a first input feature corresponding to the first video frame, and to fuse the second mask area and the second video frame feature into a second input feature corresponding to the second video frame; and to determine the temporal feature and the spatial feature of the local information based on the first input feature and the second input feature.

[0017] In some embodiments of the present application, the feature construction module is also used to subtract the first input feature from the second input feature to obtain a difference feature; obtain the time series feature by reducing the dimension of the difference feature; reduce the dimension of the first input feature to obtain a first reduced dimension feature, and reduce the dimension of the second input feature to obtain a second reduced dimension feature; and perform differential processing on the first reduced dimension feature and the second reduced dimension feature to obtain the spatial feature.

[0018] In some embodiments of the present application, the difference feature includes: a difference feature map; the feature construction module is further used to perform a pooling operation on the difference feature map in a preset direction to obtain a pooling feature map; the pooling feature map is split according to the preset direction to obtain N sub-feature maps; wherein N is the length of the pooling feature map in the preset direction, and N is greater than or equal to 1; pooling operations are performed on the N sub-feature maps respectively to obtain pooling features corresponding to each of the N sub-feature maps; and the pooling features corresponding to each of the N sub-feature maps are used to splice into the time series feature.

[0019] In some embodiments of the present application, the similarity determination module is further used to fuse the temporal features and the spatial features to obtain spatiotemporal fusion features; and perform similarity recognition on the spatiotemporal fusion features through a similarity judgment model to obtain the text similarity between the first video frame and the second video frame.

[0020] In some embodiments of the present application, the similarity determination module is further used to determine the first video frame and the second video frame as the key frames corresponding to the local information in the video to be detected when the text similarity is less than a similarity threshold; and when the text similarity is greater than or equal to the similarity threshold, determine the first video frame or the second video frame as the key frame corresponding to the local information in the video to be detected.

[0021] In some embodiments of the present application, the similarity determination module is further used to, when the text similarity is greater than or equal to the similarity threshold, fuse the first video frame and the second video frame to obtain a fused video frame; and determine the fused video frame as the key frame corresponding to the local information in the video to be detected.

[0022] In some embodiments of the present application, the feature construction module is also used to extract features from the first video frame features to obtain first features to be reconstructed; reconstruct the first features to be reconstructed by increasing the dimension to obtain first reconstructed features; fuse the first fused features based on the first video frame features and the first reconstructed features; and fuse the feature components of each channel in the first fused features to obtain the first mask area representing the position of the local information in the first video frame.

[0023] In some embodiments of the present application, the first fusion feature includes: a first fusion feature map, and the feature component includes: a sub-fusion feature map; the feature construction module is further used to generate a channel feature vector for each channel based on the sub-fusion feature map of each channel of the first fusion feature map; a corresponding attention weight is calculated for the channel feature vector of each channel; based on the attention weight, the channel feature vector of each channel is weightedly fused to obtain a fusion feature vector; the feature map is restored for the fusion feature vector to obtain the first mask area representing the position of the local information in the first video frame.

[0024] In some embodiments of the present application, the feature extraction module is further used to divide the first video frame and the second video frame into blocks respectively to obtain multiple first image blocks and multiple second image blocks; perform convolution processing on the multiple first image blocks respectively to obtain multiple first convolution feature maps corresponding to the multiple first image blocks; perform convolution processing on the multiple second image blocks respectively to obtain multiple second convolution feature maps corresponding to the multiple second image blocks; determine the multiple first convolution feature maps as the first video frame features, and determine the multiple second convolution feature maps as the second video frame features.

[0025] The present invention provides a key frame detection device, including:

[0026] a memory for storing executable instructions;

[0027] The processor is configured to implement the key frame detection method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0028] An embodiment of the present application provides a computer-readable storage medium including executable instructions. When the executable instructions are executed by a processor, a key frame detection method provided in the embodiment of the present application is implemented.

[0029] An embodiment of the present application provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, the key frame detection method provided in the embodiment of the present application is provided.

[0030] The embodiments of the present application have the following beneficial effects: the key frame detection device will simultaneously determine the temporal features characterizing the position change of local information between the first video frame and the second video frame, as well as the spatial features characterizing the appearance of the local information in the first video frame and the second video frame, and further combine the features in the two dimensions of position change and appearance to derive information similarity, so that the features considered when determining the similarity are more comprehensive, thereby improving the accuracy of judging the information similarity and ultimately improving the accuracy of key frame detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the process of determining key frames through dictionary learning and sparse coding;

[0032] Figure 2 This is a schematic diagram of the key frame determination process based on the reward function of diversity and expressiveness;

[0033] Figure 3 It is a flowchart of obtaining key frames using the constructed attention-based encoding and decoding network;

[0034] Figure 4 This is a schematic diagram of the key frame detection process based on the text twin network;

[0035] Figure 5 Schematic diagram of the architecture of the key frame detection system provided in an embodiment of the present application;

[0036] Figure 6 This embodiment of the present application provides Figure 5 A schematic diagram of the server structure in FIG;

[0037] Figure 7 This is a flowchart of the key frame detection method provided in the embodiment of the present application. Figure 1 ;

[0038] Figure 8 This is a flowchart of the key frame detection method provided in the embodiment of the present application. Figure 2 ;

[0039] Figure 9 This is a flowchart of the key frame detection method provided in the embodiment of the present application. Figure 3 ;

[0040] Figure 10 Schematic diagram of the architecture of the key frame detection model provided in the embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0042] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0045] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0046] 1) Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field of computer science that seeks to understand the essence of AI and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0047] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0048] 2) Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects, and further processing them to produce images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, and smart transportation. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0049] 3) Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0050] 4) Deep learning (DL) involves learning the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to acquire the same analytical and learning capabilities as humans, enabling them to recognize data such as text, images, and sounds.

[0051] 5) Keyframes are frames that summarize the scene events in a video. Since many video frames are often irrelevant to the scene events and are redundant, using keyframes can significantly reduce the computational workload when performing processing such as video content understanding.

[0052] 6) The Attention Mechanism is a data processing method in machine learning, widely used in various machine learning tasks such as natural language processing, image recognition, and speech recognition. The attention mechanism assigns different weights to different parts of the input data, focusing the model's attention on more important information during calculations, allowing the model to make more accurate judgments.

[0053] 7) Cloud computing refers to the delivery and usage model of IT infrastructure, enabling on-demand, scalable access to required resources over the internet. In a broader sense, cloud computing refers to the delivery and usage model of services, enabling on-demand, scalable access to required services over the internet. These services can be IT-related, software-related, internet-related, or other services. Cloud computing is the product of the convergence of traditional computer and network technologies, including grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0054] Cloud computing has rapidly grown, driven by the internet, real-time data streams, the diversification of connected devices, and the growing demand for search services, social networks, mobile commerce, and open collaboration. Unlike previous parallel and distributed computing approaches, the emergence of cloud computing will fundamentally revolutionize the entire internet and enterprise management model.

[0055] With the advancement of AI research and technology, it has been applied in a variety of fields, including smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robotics, smart healthcare, and smart customer service. Keyframe detection in videos is a key area of ​​AI research. The following describes the application of AI in keyframe detection.

[0056] Keyframe detection refers to the process of selecting a small number of representative video frames or video segments from a video. The selected video frames or video segments can fully express the entire content of the video and remove redundancy. Therefore, it can be widely used in scenarios such as video understanding and video infringement analysis.

[0057] Early keyframe detection was often achieved using shallow machine learning. For example, keyframe detection was considered a subset selection problem, with submodule optimization algorithms employed to find the optimal solution to the problem, i.e., the keyframes. Alternatively, keyframes were detected using target gaze tracking techniques, in which case submodule optimization algorithms were also employed to ensure the relevance and diversity of the detected keyframes.

[0058] With the advancement of technology, there has been research on improving the information and relevance of key frames through dictionary learning and sparse coding, and extracting key frames by focusing on the motion areas and relevance between different video segments. For example, Figure 1 This is a flowchart of determining key frames through dictionary learning and sparse coding. Figure 1 , for the input video 1-1, motion region detection processing 1-2 is performed, and the motion region detection result is preprocessed 1-3 to obtain spatiotemporal features 1-4. Then, the spatiotemporal features 1-4 are sparsely represented to obtain sparse features 1-5. For the training video 1-6, the sparse features 1-5 are combined with the sparse coefficients 1-7, the dictionary features 1-8 and the dictionary graph features 1-9 to calculate the matching degree 1-10. When the matching degree 1-10 is greater than the threshold, the dictionary graph features are updated and the sparse coefficients 1-11 are recalculated, and then the key frame 1-12 is output. When the matching degree 1-10 is less than the threshold, the key frame 1-12 is directly output. For the test video 1-13, the sparse features 1-5 are combined with the spatiotemporal features. Figure 1-1 4 performs matching and updates dictionary graph features 1-9 when there is no match.

[0059] However, the feature representation and generalization capabilities of shallow machine learning are always limited, and it is impossible to achieve good key frame detection results.

[0060] Based on this, key frame detection based on deep learning has gradually become a mainstream technology. For example, by setting a reward function that considers the diversity and expressiveness of key frames, supervised and unsupervised key frame detection can be achieved. For example, Figure 2 Figure 2 is a flowchart of a keyframe determination process based on a reward function for diversity and expressiveness. First, the video is decoded into multiple video sequence sets V1-Vm. Feature extraction is then performed on these video sequence sets V1-Vm using a cascaded convolutional neural network 2-1 and a bidirectional recurrent network 2-2. Based on the extracted features, a reward function 2-3 for the diversity and expressiveness of the keyframes is generated. The reward function 2-3 and the extracted features are then combined to determine a keyframe 2-4.

[0061] For example, by reducing key frame detection to a sequence-to-sequence learning problem, a long short-term memory (LSTM) network is introduced to construct an attention-based encoding and decoding network to obtain key frames. Figure 3It is a flowchart of obtaining key frames using the constructed attention-based encoding and decoding network. At this time, the video frame sequence 3-1 of the video is first input into the encoder 3-2 for encoding, and then the weight information 3-31 determined by the attention mechanism 3-3 and the encoding result of the encoder 3-2 are input into the decoder 3-4 for decoding. Finally, the decoding result 3-5 output by the decoder 3-4 is combined with the video frame sequence 3-1 to obtain the key frame 3-6.

[0062] For example, through the supervised learning strategy, LSTM and determinant point process are used to automatically detect key frames or key segments in the video; the expanded timing units in the video are reconstructed through LSTM, and the key frame detection of the video is achieved through the reconstruction error, wherein the expanded timing units are jointly generated by considering the features of video segments of different lengths.

[0063] However, in real-world applications, many video processing methods rely on local information within the video, such as text, for keyframe detection. In this case, it is necessary to detect keyframes with different local information within the video. However, the aforementioned methods all perform keyframe detection based on the overall image content of the video, which is not ideal for detecting keyframes based on local information.

[0064] In this regard, some methods for key frame detection based on local information have emerged in related technologies. For example, key frame detection based on text twin networks calculates the similarity of adjacent video frames through the twin network, and introduces an attention module to recognize the text in the video frame, thereby determining the key frame based on the recognized text. For example, Figure 4 This is a schematic diagram of the process of key frame detection based on the text twin network. For the adjacent video frame 4-2 in the video 4-1, the text twin network, namely network 4-31 and network 4-32 (wherein, network 4-31 and network 4-32 respectively contain convolutional layers, attention modules and composite modules (modules composed of multiple convolutional layers), and network 4-31 and network 4-32 parameters share 4-4) are respectively extracted for features, and then the extracted features are integrated into the fully connected layer 4-5 for recognition to obtain similarity 4-6, thereby obtaining key frames 4-8. At the same time, the attention modules of network 4-31 and network 4-32 will also output the results of whether the image contains text 4-71 or does not contain text 4-72.

[0065] Although key frame detection based on the text twin network can detect text key frames in the video, this method uses all the content of the video frame to analyze the similarity of local information. Therefore, when the local information in the video frame, that is, the text, remains unchanged, while other content of the video frame changes significantly, this method cannot effectively extract the key frames, resulting in a high repetition rate of key frames.

[0066] In order to improve the accuracy of key frame detection, a method of performing key frame detection based on the position of local information has also emerged in the related art, for example, a method of performing key frame detection based on the position of text blocks. This method is based on a text block mask network, which automatically calculates the position of text blocks in video frames. By combining the text block position with the feature map extracted by the network, the similarity between adjacent frames can be effectively calculated through the text block position. Then, based on the similarity between adjacent frames, key frames are determined from the video, for example, frames with higher similarity are discarded and frames with lower similarity are retained. However, the repetition rate and recall rate of text key frames generated by this method at any sampling rate are unstable. The reason for this is that when calculating similarity based on the text block position, the accuracy of similarity calculation is low, which affects the accuracy of key frame detection.

[0067] As can be seen from the above, in the related art, when determining key frames based on local information, there is a problem of low accuracy in calculating the similarity of local information, which results in low accuracy in key frame detection.

[0068] Furthermore, the positioning accuracy of local information can also affect the accuracy of keyframe detection. In related technologies, there is a certain amount of error when positioning local information, such as when locating text blocks. This inevitably leads to large errors in similarity, ultimately reducing the accuracy of keyframe detection.

[0069] The embodiments of the present application provide a key frame detection method, apparatus, device, computer-readable storage medium, and program product, which can improve the accuracy of key frame detection. The following describes an exemplary application of the key frame detection device provided by the embodiments of the present application. The key frame detection device provided by the embodiments of the present application can be implemented as various types of terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as a server. Below, an exemplary application of the key frame detection device when it is implemented as a server will be described.

[0070] See also Figure 5 , Figure 5This is an architectural diagram of the key frame detection system provided in an embodiment of the present application. To support a key frame detection application, the terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0071] The terminal 400 is used to respond to the user's operation on the graphical interface 400 - 1 , generate a video to be detected, and send the video to be detected to the server 200 via the network 300 .

[0072] The server 200 is used to extract features from the first video frame and the second video frame of the video to be detected, respectively, to obtain first video frame features of the first video frame and second video frame features of the second video frame; based on the first video frame features and the second video frame features, a temporal feature and a spatial feature are constructed; wherein the temporal feature represents the position change of local information in the video to be detected between the first video frame and the second video frame, and the spatial feature represents the appearance of the local information in the first video frame and the second video frame; using the temporal feature and the spatial feature, the information similarity between the first video frame and the second video frame is determined; based on the information similarity, the key frame corresponding to the local information in the video to be detected is determined.

[0073] The server 200 is further configured to send the key frame to the terminal 400 , and the terminal 400 displays the key frame on the graphical interface 400 - 1 .

[0074] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart home appliance, car terminal, etc., but is not limited to these. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present invention.

[0075] See also Figure 6 , Figure 6 This embodiment of the present application provides Figure 5 The structural diagram of the server in Figure 2The server 200 shown includes: at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 6 Various buses are labeled as bus system 240 .

[0076] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0077] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0078] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0079] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0080] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0081] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0082] A network communication module 252 for reaching other computing devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0083] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0084] The input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more input devices 232 and to translate the detected inputs or interactions.

[0085] In some embodiments, the key frame detection device provided in the embodiments of the present application can be implemented in a software manner. Figure 6 A keyframe detection device 255 stored in memory 250 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a feature extraction module 2551, a feature construction module 2552, a similarity determination module 2553, and a keyframe determination module 2554. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0086] In other embodiments, the key frame detection device provided in the embodiments of the present application can be implemented in hardware. As an example, the key frame detection device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the key frame detection method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0087] In some embodiments, the terminal or server can implement the permission issuance method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a key frame detection APP; it can also be a small program, that is, a program that can be run only by downloading it into a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in.

[0088] The embodiments of the present application can be applied to: Below, the key frame detection method provided by the embodiments of the present application will be described in combination with the exemplary application and implementation of the key frame detection device provided by the embodiments of the present application.

[0089] See also Figure 7 , Figure 7 This is a flowchart of the key frame detection method provided in the embodiment of the present application. Figure 1 , will combine Figure 7 The steps shown are explained.

[0090] S101 , performing feature extraction on a first video frame and a second video frame of a video to be detected, respectively, to obtain a first video frame feature of the first video frame and a second video frame feature of the second video frame.

[0091] The embodiment of the present application is implemented in the scenario of key frame detection of a video. First, the key frame detection device will use the video frames of the video to be detected for key frame detection as the first video frame, then determine the second video frame to be similar to the first video frame, and then perform feature extraction on the first video frame to obtain the first video frame feature, and perform feature extraction on the second video frame to obtain the second video frame feature.

[0092] It is understandable that the video to be detected can be a short video (less than 5 minutes in length) shot and uploaded by the user, such as a mashup video, a clip taken from a TV series, an MV, etc. The video to be detected can also be a long video released by a professional video producer, such as a nature documentary, a movie, a TV series, etc., which is not limited in this application.

[0093] It should be noted that the second video frame can be a video frame adjacent to the first video frame, such as the next frame of the first video frame, or it can be a video frame that differs from the first video frame by a fixed number, such as the 10th video frame before the first video frame, etc. This application does not limit this.

[0094] In some embodiments, the key frame detection device can also first downsample the video to be detected to obtain a downsampled frame sequence, and then use the video frames in the downsampled frame sequence as the first video frame one by one, and at the same time determine the corresponding second video frame for the first video frame. In this way, the number of video frames to be calculated during key frame detection can be greatly reduced.

[0095] It is understandable that the key frame detection device can achieve downsampling by extracting frames at fixed intervals, for example, extracting one or more frames per second to obtain a downsampled frame sequence; the key frame detection device can also use the video understanding technology in artificial intelligence to extract fragments that describe important semantics from the video to be detected, and use the video frames corresponding to these fragments to generate a downsampled frame sequence. This application is not limited here.

[0096] The temporal order of the first video frame and the second video frame can be set according to actual conditions. For example, the first video frame can be set to be in front, or the second video frame can be set to be in front. As long as there is a timing difference between the first video frame and the second video frame, this application does not limit it here.

[0097] In some embodiments, the key frame detection device can directly determine the image features extracted from the first video frame using the Convolutional Neural Network (CNN) model as the first video frame features, and determine the image features extracted from the second video frame using the CNN model as the second video frame features.

[0098] In other embodiments, the key frame detection device can also perform block processing on the first video frame and the second video frame, and then extract image features for the image blocks obtained by the block processing, and use the image features of each image block of the first video frame to generate the first video frame features, and use the image features of each image block of the second video frame to generate the second video frame features.

[0099] S102: Constructing temporal features and spatial features based on the first video frame features and the second video frame features.

[0100] The key frame detection device determines the position change of the local information of the video to be detected in the first video frame and the second video frame based on the obtained first video frame features and second video frame features, such as the position change of the icon in the first video frame and the second video frame, the text in the text information that changes in the first video frame and the second video frame, etc. Since there is a temporal difference between the first video frame and the second video frame, the change of the local information in the first video frame and the second video frame is often a change in the temporal dimension, and the key frame detection device will determine the change as a temporal feature. At the same time, the key frame detection device will also use the first video frame features and the second video frame features to determine the appearance of the local information in the first video frame and the second video frame, such as the content of the text information, the color and shape of the image information, etc. Since the appearance is generally reflected in space, the features obtained based on the appearance can be used as spatial features.

[0101] That is to say, in the embodiment of the present application, the temporal feature represents the movement of the local information in the video to be detected between the first video frame and the second video frame, and the spatial feature represents the appearance of the local information in the first video frame and the second video frame.

[0102] It is understandable that local information may refer to text information (text content is not limited) appearing in the video to be detected, or may refer to image information of special icons or special characters appearing in the video to be detected, which is not limited in this application.

[0103] In some embodiments, the key frame detection device identifies the dimensionality reduction features of the first video frame features and the second video frame features to preliminarily locate the mask areas representing the positions of the local information in the first video frame and the second video frame, respectively. Then, based on the mask areas, the feature extraction attention is focused on the features related to the local information in the first video frame features and the second video frame features. Then, based on the features related to the local information in the first video frame features and the features related to the local information in the second video frame features, the temporal features and spatial features are determined.

[0104] In other embodiments, the key frame detection device may also first use the features of the template of local information to match the first video frame features and the second video frame features, use the features directly matched from the first video frame features and the second video frame features as spatial features, and subtract the features matched from the first video frame features and the features matched from the second video frame features to obtain temporal features.

[0105] S103: Determine information similarity between the first video frame and the second video frame using temporal features and spatial features.

[0106] After obtaining the temporal and spatial features, the keyframe detection device then identifies similarities based on these features, obtaining information similarity that characterizes the similarities between the local information in the first and second video frames. This information similarity takes into account both the positional changes of the local information between the first and second video frames—that is, the dynamic changes in the local information—as well as the appearance of the local information in the first and second video frames, thus taking into account a more comprehensive range of feature dimensions.

[0107] In some embodiments, the key frame detection device can directly input the temporal features and spatial features into a trained machine learning or deep learning model to calculate information similarity.

[0108] In other embodiments, the key frame detection device can also match the temporal features and spatial features with preset features of different similarity scores (or levels), thereby determining the similarity score hit by the temporal features and the similarity score hit by the spatial features, and determining the average of these two similarity scores as the final information similarity.

[0109] S104: Determine the key frame corresponding to the local information in the video to be detected based on the information similarity.

[0110] After determining the information similarity, the key frame detection device can determine whether the first video frame and the second video frame depict the same or similar content based on the information similarity, thereby determining whether to designate the first video frame or the second video frame as a key frame or to discard the first video frame or the second video frame. In this way, the key frame detection device can determine the key frames of the video to be detected.

[0111] It should be noted that when the local information is different, the key frame determined from the video frame to be detected will also be different. For example, when the local information is text, the key frame is a video frame containing different text. When the local information is a preset icon, the key frame is a video frame containing the preset icon. In other words, in this application, the key frame determined by the key frame detection device corresponds to the local information.

[0112] It can be understood that compared with the method of determining key frames based on local information in related technologies, in the embodiment of the present application, the key frame detection device will simultaneously determine the temporal characteristics that characterize the position change of local information between the first video frame and the second video frame, as well as the spatial characteristics that characterize the appearance of the local information in the first video frame and the second video frame, and further combine the characteristics in the two dimensions of position change and appearance to derive information similarity, so that the features considered when determining the similarity are more comprehensive, thereby improving the accuracy of judging the information similarity and ultimately improving the accuracy of key frame detection.

[0113] based on Figure 7 , see Figure 8 , Figure 8 This is a flowchart of the key frame detection method provided in the embodiment of the present application. Figure 2 In some embodiments of the present application, based on the first video frame feature and the second video frame feature, the temporal feature and the spatial feature are constructed, that is, the specific implementation process of S102 may include: S1021-S1023, as follows:

[0114] S1021. Determine a first mask region representing a position of local information in the first video frame based on the first video frame feature, and determine a second mask region representing a position of local information in the second video frame based on the second video frame feature.

[0115] The key frame detection device first uses the first video frame features to locate local information from the first video frame, and then determines the attention weight for each feature in the first video frame features based on the position of the local information, so as to characterize the position of the local information through different attention weights. For example, for the features corresponding to the local information, a higher weight is determined, and for the features corresponding to other parts, a lower weight is determined. Next, the key frame detection device uses the attention weights corresponding to different features to combine into a feature map, thereby obtaining the first mask area. The key frame detection device can determine the second mask area in the same way.

[0116] S1022: Fusing the first mask region and the first video frame feature into a first input feature corresponding to the first video frame, and fusing the second mask region and the second video frame feature into a second input feature corresponding to the second video frame.

[0117] The key frame detection device fuses the first mask area with the first video frame feature so that the features corresponding to the local information in the first video frame feature are retained, and the features corresponding to content other than the local information in the first video frame feature are suppressed, so that when the first input feature is subsequently processed, the processing attention is focused on the features corresponding to the local information in the first video frame feature. Similarly, the key frame detection device fuses the second mask area with the second video frame feature so that when the second input feature is subsequently processed, the processing attention is focused on the features corresponding to the local information in the second video frame feature.

[0118] In some embodiments, the key frame detection device may obtain the first input feature by weighting the first mask region and the first video frame feature. In other embodiments, the key frame detection device may also directly superimpose the first mask region on the first video frame feature to obtain the first input feature.

[0119] It is understandable that the key frame detection device will use the same fusion method as the first input feature to fuse and generate the second input feature.

[0120] S1023: Determine temporal features and spatial features of the local information based on the first input feature and the second input feature.

[0121] After obtaining the first input feature and the second input feature, the key frame detection device can obtain a time series feature by directly subtracting the first input feature and the second input feature, directly extract features from the first input feature and the second input feature respectively, and use the extracted features to fuse into a spatial feature, or extract features from the difference between the first input feature and the second input feature to obtain a time series feature, and use the features extracted from the first input feature and the second input feature to subtract to obtain a spatial feature. This application is not limited here.

[0122] In an embodiment of the present application, the key frame detection device can first determine the first mask area and the second mask area based on the first video frame features and the second video frame features, respectively, and then fuse the first video frame features with the first mask area, and fuse the second video frame features with the second mask area, to obtain features corresponding to the image content that is irrelevant to local information in the first video features and the second video frame features, thereby ensuring that when the temporal features and spatial features are subsequently generated, they are performed on local information, thereby ensuring the accuracy of the subsequently calculated information similarity.

[0123] In some embodiments of the present application, determining the temporal features and spatial features of the local information based on the first input feature and the second input feature, that is, the specific implementation process of S1023, may include: S1023a-S1023d, as follows:

[0124] S1023a: Subtract the first input feature from the second input feature to obtain a difference feature.

[0125] It is understandable that the key frame detection device can use the first input feature to subtract the second input feature to obtain the difference feature, or use the second input feature to subtract the first input feature to obtain the difference feature. This application does not limit this.

[0126] S1023b. Obtain time series features by performing dimensionality reduction on the difference features.

[0127] The key frame detection device can achieve dimensionality reduction of the difference features by performing channel compression on the difference features, or by performing pooling operations on the difference features in a specific direction, such as a preset direction, to achieve dimensionality reduction of the difference features. It can also first perform pooling operations on the difference features in a specific direction, and then splice the obtained pooled features to achieve dimensionality reduction of the difference features to obtain time series features. This application is not limited here.

[0128] S1023c. Perform dimensionality reduction on the first input feature to obtain a first reduced dimensionality feature, and perform dimensionality reduction on the second input feature to obtain a second reduced dimensionality feature.

[0129] It can be understood that the dimensionality reduction process of the first input feature and the second input feature is similar to the dimensionality reduction process of the difference feature, and will not be repeated here.

[0130] S1023d. Perform differential processing on the first dimensionality reduction feature and the second dimensionality reduction feature to obtain a spatial feature.

[0131] The key frame detection device performs differential processing on the first dimensionality reduction feature and the second dimensionality reduction feature by performing a difference between the first dimensionality reduction feature and the second dimensionality reduction feature. The obtained difference result is the spatial feature.

[0132] It should be noted that the order in which temporal features and spatial features are generated does not affect the final information similarity. Therefore, in other embodiments, the key frame detection device may first execute S1023c-S1023d and then S1023a-S1023b, or may execute S1023a-S1023b and S1023c-S1023d simultaneously, which is not limited in this application.

[0133] In some embodiments of the present application, the difference feature includes a difference feature graph. In this case, the time series feature is obtained by performing dimensionality reduction on the difference feature, that is, the specific implementation process of S1023b may include S201-S204 as follows:

[0134] S201. Perform a pooling operation on the difference feature map in a preset direction to obtain a pooled feature map.

[0135] The key frame detection device performs a pooling operation on the difference feature map in a preset direction so that the features of the difference feature map in the preset direction are merged into one feature to generate a new feature map, which is the pooled feature map.

[0136] It is understandable that the preset direction can be the horizontal direction, that is, the direction of the width of the difference feature map, or the vertical direction, that is, the direction of the height of the difference feature map. In this way, the resulting pooled feature map is either a feature map with a height equal to the height of the difference feature map and a width of 1, or a feature map with a height of 1 and a width equal to the width of the difference feature map.

[0137] It should be noted that the number of channels of the difference feature map may not be 1. When the key frame detection device pools the feature map of each channel in a preset direction, the channels are not merged, so the number of channels of the obtained pooled feature map is the same as the number of channels of the difference feature map.

[0138] Exemplarily, when the difference feature map is 64*56*56 (C*H*W) and the preset direction is the horizontal direction, the key frame detection device generates a pooled feature map of 64*56*1.

[0139] S202: Split the pooled feature map according to a preset direction to obtain N sub-feature maps.

[0140] Next, the key frame detection device will split the pooled feature map in the preset direction to obtain N sub-feature maps. When the preset direction is vertical, the key frame detection device will obtain sub-feature maps of the width of the difference feature map, and when the preset direction is horizontal, the key frame detection device will obtain feature maps of the height of the pooled feature map. In other words, the key frame detection device will obtain sub-feature maps of the length of the pooled feature map in the preset direction, so N is the length of the difference feature map in the preset direction. N is greater than or equal to 1.

[0141] It can be understood that the size of each sub-feature map is C*1*1, where C is the number of channels of the pooled feature map and also the number of channels of the difference feature map.

[0142] For example, for a 64*56*1 pooled feature map, the key frame detection device will obtain 56 64*1*1 sub-feature maps when splitting it horizontally.

[0143] S203: Perform pooling operations on the N sub-feature maps respectively to obtain pooling features corresponding to the N sub-feature maps.

[0144] After obtaining N sub-feature maps, the key frame detection device performs a pooling operation on each sub-feature map, that is, each sub-feature map is combined into a single feature value. This feature is the pooled feature corresponding to each sub-feature. In this way, the key frame detection device will obtain N pooled features.

[0145] It can be understood that the pooling operation in the embodiment of the present application may refer to a maximum pooling operation or an average pooling operation, and the present application does not limit this.

[0146] S204: Use the pooled features corresponding to the N sub-feature maps to splice into a time series feature.

[0147] Finally, the keyframe detection device will be combined with N sub-features Figure 1 The corresponding N pooled features are concatenated to obtain a feature vector. This feature vector is the time series feature to be determined.

[0148] In some embodiments, the key frame detection device may splice the N pooled features in the order of the N sub-feature maps. In other embodiments, the key frame detection device may also splice the N time features in a random order, which is not limited in this application.

[0149] In an embodiment of the present application, the key frame detection device performs preliminary dimensionality reduction on the difference feature map through a pooling operation, and then splits it into sub-feature maps. The pooling operation is continued on the sub-feature maps to preserve some of the most representative features in the difference feature map, that is, to obtain pooled features. Finally, the temporal features based on the pooled features are used to facilitate the subsequent calculation of information similarity.

[0150] In some embodiments of the present application, the first input feature includes: a first input feature map; and the specific implementation process of performing dimensionality reduction on the first input feature to obtain the first reduced dimensionality feature, i.e., S1023c, may include: S205-S208, as follows:

[0151] S205. Perform a pooling operation on the first input feature map in a preset direction to obtain a first pooling map.

[0152] S206 : Split the first pooling map along a preset direction to obtain M first sub-pooling maps, where M is the length of the first pooling map along the preset direction, and M is greater than or equal to 1.

[0153] S207 , performing pooling operations on the M first sub-pooling maps respectively to obtain output features corresponding to each of the M first sub-pooling maps.

[0154] S208. Determine the output features corresponding to the M first sub-pooling maps as first dimensionality reduction features.

[0155] It should be noted that the processing of S205-S207 is basically similar to the processing of S201-S203, and will not be repeated here.

[0156] In some embodiments of the present application, the second input feature includes: a second input feature map; and the specific implementation process of performing dimensionality reduction on the second input feature to obtain the second reduced dimensionality feature, i.e., S1023c, may include: S209-S212, as follows:

[0157] S209: Perform a pooling operation on the second input feature map in a preset direction to obtain a second pooling map.

[0158] S210: Split the second pooling map along a preset direction to obtain L second sub-pooling maps, where L is the length of the second pooling map along the preset direction, and L is greater than or equal to 1.

[0159] S211. Perform pooling operations on the L second sub-pooling maps respectively to obtain output features corresponding to the L second sub-pooling maps.

[0160] S212. Determine the output features corresponding to the L second sub-pooling maps as second dimensionality reduction features.

[0161] It should be noted that the processing of S209-S211 is basically similar to the processing of S201-S203, and will not be repeated here.

[0162] based on Figure 7 , see Figure 9 , Figure 9 This is a flowchart of the key frame detection method provided in the embodiment of the present application. Figure 3 In some embodiments of the present application, information similarity includes text similarity. In this case, the information similarity between the first video frame and the second video frame is determined using temporal features and spatial features. The specific implementation process of S103 may include S1031-S1032 as follows:

[0163] S1031. Fuse the temporal features and spatial features to obtain spatiotemporal fusion features.

[0164] It is understandable that the key frame detection device can splice the temporal features and spatial features to obtain the temporal and spatial fusion features, or can obtain the temporal and spatial fusion features by summing the temporal features and spatial features. This application does not limit this.

[0165] S1032: Perform similarity recognition on the spatiotemporal fusion features through a similarity judgment model to obtain text similarity between the first video frame and the second video frame.

[0166] Next, the key frame detection device will input the fused spatiotemporal fusion features into the trained similarity judgment model to use the similarity judgment model to perform similarity recognition on the spatiotemporal fusion features. The recognition result obtained is the text similarity between the first video frame and the second video frame.

[0167] It should be noted that text similarity represents the degree of similarity between the subtitle text in the first video frame and the subtitle text in the second video frame. Therefore, subsequent keyframe screening is based on text similarity, thereby selecting text-based keyframes in the video to be tested, i.e., text keyframes. It is understandable that different text keyframes contain different text.

[0168] In some embodiments, the similarity judgment model is a fully connected layer, and in other embodiments, the similarity judgment model is a Softmax layer, which is not limited in this application.

[0169] In an embodiment of the present application, the key frame detection device can determine the text similarity between the first video frame and the second video frame based on temporal features and spatial features, so as to subsequently determine the text key frames in the video to be detected based on the text similarity.

[0170] In some embodiments of the present application, based on information similarity, the key frames corresponding to the local information in the video to be detected are screened out, that is, the specific implementation process of S104 may include: S1041 or S1042 as follows:

[0171] S1041: When the text similarity is less than the similarity threshold, both the first video frame and the second video frame are determined as key frames corresponding to the local information in the video to be detected.

[0172] The key frame detection device compares the text similarity with the similarity threshold. When it is determined that the text similarity is less than the similarity threshold, it is considered that the text contained in the first video frame and the second video frame is different. At this time, the first video frame and the second video frame will be determined as the key frames corresponding to the local information.

[0173] S1042: When the text similarity is greater than or equal to the similarity threshold, determine the first video frame or the second video frame as a key frame corresponding to the local information in the video to be detected.

[0174] When the text similarity is greater than or equal to the similarity threshold, the keyframe detection device will determine that the text contained in the first and second video frames is the same. Even if the content of the first and second video frames is different, the keyframe detection device will only retain one of the first and second video frames. In this way, the text in the resulting keyframes will be different.

[0175] It is understandable that the similarity threshold can be set according to actual conditions, or it can be dynamically adjusted according to the category of the video to be detected. For example, when the video to be detected is a TV series, the similarity threshold is adjusted to 0.5, or when the video to be detected is a short video, the similarity threshold is adjusted to 0.8, etc. This application does not limit this.

[0176] In an embodiment of the present application, the key frame detection device can filter out the key frames of the video to be detected from the first video frame and the second video frame based on the size relationship between the text similarity and the similarity threshold, so that the video frames containing different texts in the video frames to be detected are determined as key frames, reducing the redundancy of key frames.

[0177] In some embodiments of the present application, the specific implementation process of S104 may further include: S1043-S1044, as follows:

[0178] S1043: When the text similarity is greater than or equal to the similarity threshold, the first video frame and the second video frame are fused to obtain a fused video frame.

[0179] S1044: Determine the fused video frame as a key frame corresponding to the local information in the video to be detected.

[0180] When the text similarity exceeds a similarity threshold, the keyframe detection device can select one of the first and second video frames as a keyframe and fuse the first and second video frames into a single video frame, thereby generating a fused video frame. Finally, the keyframe detection device determines the fused video frame as the keyframe corresponding to the local information, so that the image content of both the first and second video frames is preserved in the keyframe.

[0181] In an embodiment of the present application, the key frame detection device can fuse the first video frame and the second video frame when the texts of the first video frame and the second video frame are relatively similar, so that more image information is retained in the obtained key frame, making the key frame have a larger amount of information.

[0182] In some embodiments of the present application, determining a first mask region representing a location of local information in the first video frame based on features of the first video frame, i.e., a specific implementation process of S1021, may include: S1021a-S1021d, as follows:

[0183] S1021a: Extract features from the first video frame to obtain first features to be reconstructed.

[0184] The key frame detection device inputs the first video frame feature into the feature extraction model, so that the feature extraction model further reduces the dimension of the first video frame feature to obtain the first feature to be reconstructed. It is understood that the feature extraction model may include one or more convolutional layers.

[0185] When the feature extraction model includes only one convolution layer, the key frame detection device uses the output of the convolution layer as the first feature to be reconstructed.

[0186] When the feature extraction model includes multiple cascaded convolutional layers (each convolutional layer outputs an intermediate feature), the key frame detection device will determine the intermediate features output by each convolutional layer as the first feature to be reconstructed.

[0187] S1021b: reconstruct the first feature to be reconstructed by increasing its dimension to obtain a first reconstructed feature.

[0188] After obtaining the first feature to be reconstructed, the key frame detection device can perform dimensionality-increasing reconstruction on the first feature to be reconstructed through deconvolution or upsampling, so that the first feature to be reconstructed can be transformed into a first reconstructed feature with a higher dimension. The following uses deconvolution as an example to illustrate the dimensionality-increasing reconstruction process.

[0189] When the feature extraction model includes only one convolution layer, the key frame detection device directly uses one deconvolution to obtain the first reconstructed feature.

[0190] When the feature extraction model includes multiple cascaded convolutional layers, the key frame detection device first deconvolves the intermediate features output by the last convolutional layer, merges the result with the intermediate features output by the penultimate convolutional layer to obtain a merged feature, and then continues to deconvolve the merged feature. This iterative process is repeated until the number of deconvolution layers is completed, and the result obtained from each deconvolution is determined as the first reconstructed feature.

[0191] It can be understood that by reconstructing the first feature to be reconstructed through increased dimensionality, local information in the video frame can be located more effectively, so that the position representation of the local information is more accurate, thereby further improving the accuracy of similarity calculation.

[0192] S1021c: Fusing a first fusion feature based on the first video frame feature and the first reconstruction feature.

[0193] In an embodiment of the present application, the key frame detection device fuses one or more deconvolution results included in the first reconstruction feature based on the first video frame feature to obtain a first fused feature.

[0194] In other embodiments, the key frame detection device can also fuse the first video frame feature with the last deconvolution result in the first reconstructed feature to obtain a temporary fusion result, and at the same time splice the merged feature obtained in multiple deconvolution processes with the temporary fusion result to obtain the final first fusion feature.

[0195] S1021d. Fusing the feature components of each channel in the first fused feature to obtain a first mask region representing the position of the local information in the first video frame.

[0196] The first fusion feature contains feature components of multiple channels. At this time, the key frame detection device can fuse the feature components of each channel in the first fusion feature through the attention mechanism, so as to clarify in which area of ​​the first video frame the local information appears, and generate attention weights for the features of the local information and the features of other content respectively, and finally use these attention weights to form the first mask area. In this way, the area of ​​the local information in the first video frame can be represented by the difference in attention weights.

[0197] In some embodiments of the present application, determining a second mask region representing a position of local information in the second video frame based on the second video frame feature, i.e., a specific implementation process of S1021, may further include S1021e-S1021h as follows:

[0198] S1021e: Extract features of the second video frame to obtain second features to be reconstructed.

[0199] S1021f, reconstructing the second feature to be reconstructed by increasing the dimension to obtain a second reconstructed feature.

[0200] S1021g: Fuse the second fusion feature based on the second video frame feature and the second reconstruction feature to generate a second fusion feature.

[0201] S1021h: Fuse the feature components of each channel in the second fused feature to obtain a second mask region representing the position of the local information in the second video frame.

[0202] It should be noted that the processing of S1021e-S1021h is similar to the processing of S1021a-S1021d, and will not be repeated here.

[0203] In some embodiments of the present application, the first fused feature includes: a first fused feature map, and the feature components include: a sub-fused feature map; the feature components of each channel in the first fused feature are fused to obtain a first mask area representing the position of the local information in the first video frame, that is, the specific implementation process of S1021d may include: S301-S304, as follows:

[0204] S301. Generate a channel feature vector corresponding to each channel based on the sub-fusion feature map of each channel of the first fusion feature map.

[0205] The sub-fusion feature map of each channel contains multiple eigenvalues ​​(the number of eigenvalues ​​is the product of the height and width of the sub-fusion feature map), and then these eigenvalues ​​are spliced ​​into a feature vector, which is the channel feature vector corresponding to each channel.

[0206] For example, when the first fused feature map is 64*56*56 (C*H*W), the key frame detection device pulls the 56*56 feature map into a 3136-dimensional vector, thereby obtaining 64 3136-dimensional vectors.

[0207] S302: Calculate the corresponding attention weight for the channel feature vector of each channel.

[0208] When the key frame detection device obtains the channel feature vectors of each channel, it will calculate the attention vector for these channel feature vectors. Specifically, the key frame detection device will input the channel feature vectors into the channel attention model, use the parameters of the channel attention model to first calculate the encoding vector, and then perform normalization calculation on the encoding vector. The normalized result is the attention weight. This process is shown in Equation (1)-Equation (2):

[0209] e i =W i ·f i +b i (1)

[0210]

[0211] Among them, f i is the channel feature vector for each channel, W i is the weight parameter of the channel attention model, b i is the bias parameter of the channel attention model, e is the encoding vector, α i is the normalized result.

[0212] S303: Based on the attention weight, perform weighted fusion on the channel feature vectors of each channel to obtain a fused feature vector.

[0213] Then, the key frame detection device uses the attention weight as the weight of the channel feature vector of each channel in the weighting, and performs weighted fusion to obtain the fused feature vector. This process can be shown as formula (3):

[0214] f attn =α i ·f i (3)

[0215] Among them, f attn is the attention fusion feature vector.

[0216] S304 : Restoring a feature map for the fused feature vector to obtain a first mask region representing a position of the local information in the first video frame.

[0217] Finally, the key frame detection device restores the fused feature vector to a feature map. The obtained feature map is the first mask area, which indicates which areas need to be allocated more attention in subsequent processing.

[0218] In an embodiment of the present application, the key frame detection device can determine the first mask area representing the location of the local information by performing an attention mechanism on the first fusion feature, so as to subsequently determine the content that needs to be focused on when constructing the temporal features and spatial features based on the first mask area.

[0219] In some embodiments of the present application, feature extraction is performed on the first video frame and the second video frame of the video to be detected, respectively, to obtain a first video frame feature of the first video frame and a second video frame feature of the second video frame. That is, a specific implementation process of S101 may include: S1011-S1014, as follows:

[0220] S1011 . Divide the first video frame and the second video frame into blocks respectively to obtain a plurality of first image blocks and a plurality of second image blocks.

[0221] In an embodiment of the present application, in order to more accurately locate the text in the first video frame and the second video frame, the key frame detection device will respectively divide the first video frame and the second video frame into blocks, thereby obtaining multiple first image blocks and multiple second image blocks, so as to facilitate the subsequent construction of temporal features and spatial features based on smaller image blocks, thereby limiting the similarity comparison to a smaller range and improving accuracy.

[0222] It is understood that the key frame detection device can evenly divide the first video frame and the second video frame to obtain multiple first video frames of the same size and multiple second video frames of the same size. Of course, the key frame detection device can also unevenly divide the first video frame and the second video frame to obtain multiple first video frames of different sizes and multiple second video frames of different sizes.

[0223] S1012. Perform convolution processing on the multiple first image blocks respectively to obtain multiple first convolution feature maps corresponding to the multiple first image blocks.

[0224] S1013. Perform convolution processing on the multiple second image blocks respectively to obtain multiple second convolution feature maps corresponding to the multiple second image blocks.

[0225] S1014. Determine the multiple first convolution feature maps as first video frame features, and determine the multiple second convolution feature maps as second video frame features.

[0226] The key frame detection device performs convolution processing on each first image block to extract the image features contained in each first image block through convolution, thereby obtaining multiple first convolution feature maps. The key frame detection device then determines each of the multiple first convolution feature maps as first video frame features. Similarly, the key frame detection device obtains second video frame features in the same manner.

[0227] In an embodiment of the present application, the key frame detection device can obtain the first video frame features and the second video frame features by dividing the first video frame and the second video frame into blocks, and performing convolution on the multiple first image blocks and second image blocks obtained by the block division, so as to narrow the scope of constructing the temporal features and the spatial features, thereby more accurately locating the text and comparing the similarity.

[0228] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0229] The embodiments of the present application are implemented in a scenario where a server (keyframe detection device) detects text keyframes (keyframes) in a video (video to be detected), where text keyframes refer to keyframes containing different texts, regardless of the image content of these keyframes. This process can be implemented using a keyframe detection model.

[0230] First, the server decodes the video into continuous video frames, then takes two adjacent video frames to generate video frame pairs and inputs them into the input of the key frame modeling module.

[0231] Figure 10 This is a schematic diagram of the architecture of the key frame detection model provided in the embodiment of the present application. Figure 10 To illustrate the process of processing video frame pairs.

[0232] The key frame detection model includes a text block mask network 10-1 and a two-stream twin network 10-2. In the text block mask network 10-1, the Resnet18 (residual network) model is selected as the backbone network 10-11, which includes five convolutional layers: convolutional layer 10-111 to convolutional layer 10-115.

[0233] The input size of the backbone network 10-11 is 224*224. After the two video frames (the first video frame and the second video frame) are respectively input into the backbone network 10-11, they are segmented into a plurality of 56*56 image blocks (a plurality of first image blocks and a plurality of second image blocks). Specifically, unlike image segmentation for pixel-level mask prediction, the backbone network 10-11 uses the output of the convolutional layer 10-112 to obtain a 64*56*56 feature map (the first video frame feature and the second video frame feature).

[0234] In order to locate the text in the video frame, the key frame detection model will perform deconvolution on the output of the backbone network 10-11 with small features (512*7*7, 256*14*14, 128*28*28) to more accurately reconstruct the output of the convolution layer 10-113 to the convolution layer 10-115 (the first feature to be reconstructed and the second feature to be reconstructed), and use the reconstructed image (the first reconstructed feature and the second reconstructed feature) to superimpose with the output of the previous convolution layer, and uniformly input the superimposed results into the channel attention module 10-12 in the text mask network 10-1.

[0235] The channel attention module 10-12 includes a splicing layer 10-121, an attention network layer 10-122, and a mask output layer 10-123. The splicing layer 10-121 is used to splice the above-mentioned superposition results to obtain a spliced ​​feature map (first fused features and second fused features), the attention network layer 10-122 is used to weight the 64 channels of the spliced ​​feature map (fuse the feature components of each channel), and the mask output layer 10-123 is used to output a 56*56 text mask area (first mask area and second mask area).

[0236] The process of weighting the 64 channels of the concatenated feature map by the attention network layer 10-122 is as follows: the 56*56 feature map is pulled into a 3136-dimensional vector (channel feature vector), thereby generating 64 3136-dimensional features; the 64 3136-dimensional features are calculated by formula (1) and formula (2) to obtain the weights of the 64 features (attention weights), and the weighted sum is used according to formula (3) to generate the final 3136-dimensional feature (fused feature vector), and the feature vector is converted into a text mask area.

[0237] The resulting text mask area needs to be weighted with the 64*56*56 feature map output by the convolutional layer 10-12 in the backbone network 10-1, and then input into the two-stream twin network 10-2. The two-stream twin network 10-2 includes a temporal information subnetwork 10-21 and a spatial information subnetwork 10-22. The temporal information subnetwork 10-21 subtracts 10-211 the weighted feature maps (first input feature and second input feature map) corresponding to the two video frames to obtain the motion flow information (difference feature map) between adjacent frames. Then, the temporal information sub-network inputs the motion flow information into the cascaded convolutional layers, that is, convolutional layers 10-212 to 10-214, for feature extraction, and performs a pooling operation in the horizontal direction (preset direction) on the generated 64*65*65 feature map, pooling the 64*56*56 feature map into a 64*56*1 feature map, and then splitting the 64*56*1 feature map (pooled feature map) in the horizontal direction to obtain 56 64*1 feature maps (N sub-feature maps), and then performing maximum pooling or average pooling on each 64*1 feature map to obtain a local feature 10-215 (pooled feature), and then using the local features of each of the 56 feature maps to splice into a temporal fc (temporal feature).

[0238] The spatial information subnetwork 10-22 includes two branches, each consisting of convolutional layers 10-221 to 10-223. The convolutional layers of these two branches are used to extract features from the weighted feature maps of the two video frames. At the same time, the spatial information subnetwork also performs the same feature vector fc generation process as the temporal information subnetwork 10-21 on the 64*56*56 feature maps output by the two convolutional layers 10-223, thereby obtaining the feature vectors fc (first dimensionality reduction features and second dimensionality reduction features) corresponding to the two video frames respectively. These two feature vectors fc are then differentiated 10-224 (differential processing) to obtain the final spatial fc (spatial features).

[0239] Next, the key frame detection model combines the temporal fc and spatial fc, and uses the fully connected layer (similarity judgment model) to judge the text similarity of the two video frames to output text similarity (information similarity).

[0240] After obtaining the text similarity, the server determines the text keyframes based on this similarity. Specifically, when the text similarity indicates that adjacent video frames have a high degree of similarity, the two video frames are merged into one (fused video frame), or one video frame is discarded to generate the video's text keyframe (the keyframe corresponding to the local information).

[0241] It should be noted that Figure 10The block mask network 10-1 in

[10] is trained by jointly training three different tasks using a multi-task loss function. Assume that the input images during training are x1 and x2, the ground truth values ​​of whether the block location contains text are y1 and y2, and the ground truth value of the similarity between the two images is y(x1, x2). Then, the loss function for training the block mask network 10-1 can be expressed as follows:

[0242]

[0243] in, is the loss function of the text mask network, L2(·) is the L2 norm, is the output of the text block module, p(x1,x2) is the probability of similarity between two images, α is the loss weight of the text block module, set to 1, and β is the loss weight of the similarity calculation module, set to 1.

[0244] During the training of the block mask network 10-1, the Adam optimizer was used to train the network. The initial learning rate was set to 0.0005 and then reduced to 0.1 every 30 epochs. The batch size was set to 64, and the momentum and weight decay were set to 0.9 and 0.0001, respectively.

[0245] Through the above method, the time information between adjacent video frames and the spatial information of adjacent video frames can be used to more accurately measure the similarity of text in adjacent video frames. In addition, the position of the text can be accurately located to help improve the accuracy of text similarity, and ultimately achieve a higher level of detection accuracy for text key frames.

[0246] The following continues to describe the exemplary structure of the key frame detection device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 6 As shown, the software modules stored in the key frame detection device 255 of the memory 250 may include:

[0247] A feature extraction module 2551 is configured to extract features from a first video frame and a second video frame of a video to be detected, respectively, to obtain first video frame features of the first video frame and second video frame features of the second video frame;

[0248] A feature construction module 2552 is configured to construct a temporal feature and a spatial feature based on the first video frame feature and the second video frame feature; wherein the temporal feature represents a position change of local information in the video to be detected between the first video frame and the second video frame, and the spatial feature represents an appearance of the local information in the first video frame and the second video frame;

[0249] a similarity determination module 2553, configured to determine information similarity between the first video frame and the second video frame using the temporal features and the spatial features;

[0250] The key frame determination module 2554 is configured to determine the key frame corresponding to the local information in the video to be detected based on the information similarity.

[0251] In some embodiments of the present application, the feature construction module 2552 is also used to determine a first mask area representing the position of the local information in the first video frame based on the first video frame feature, and to determine a second mask area representing the position of the local information in the second video frame based on the second video frame feature; to fuse the first mask area and the first video frame feature into a first input feature corresponding to the first video frame, and to fuse the second mask area and the second video frame feature into a second input feature corresponding to the second video frame; and to determine the temporal feature and the spatial feature of the local information based on the first input feature and the second input feature.

[0252] In some embodiments of the present application, the feature construction module 2552 is also used to subtract the first input feature from the second input feature to obtain a difference feature; obtain the time series feature by reducing the dimension of the difference feature; reduce the dimension of the first input feature to obtain a first reduced dimension feature, and reduce the dimension of the second input feature to obtain a second reduced dimension feature; and perform differential processing on the first reduced dimension feature and the second reduced dimension feature to obtain the spatial feature.

[0253] In some embodiments of the present application, the difference feature includes: a difference feature map; the feature construction module 2552 is also used to perform a pooling operation on the difference feature map in a preset direction to obtain a pooling feature map; split the pooling feature map according to the preset direction to obtain N sub-feature maps; wherein N is the length of the pooling feature map in the preset direction, and N is greater than or equal to 1; perform pooling operations on the N sub-feature maps respectively to obtain pooling features corresponding to each of the N sub-feature maps; use the pooling features corresponding to each of the N sub-feature maps to splice into the time series feature.

[0254] In some embodiments of the present application, the similarity determination module 2553 is further used to fuse the temporal features and the spatial features to obtain spatiotemporal fusion features; perform similarity recognition on the spatiotemporal fusion features through a similarity judgment model to obtain the text similarity between the first video frame and the second video frame.

[0255] In some embodiments of the present application, the similarity determination module 2553 is also used to determine the first video frame and the second video frame as the key frames corresponding to the local information in the video to be detected when the text similarity is less than the similarity threshold; when the text similarity is greater than or equal to the similarity threshold, determine the first video frame or the second video frame as the key frame corresponding to the local information in the video to be detected.

[0256] In some embodiments of the present application, the similarity determination module 2553 is further used to fuse the first video frame and the second video frame to obtain a fused video frame when the text similarity is greater than or equal to the similarity threshold; and determine the fused video frame as the key frame corresponding to the local information in the video to be detected.

[0257] In some embodiments of the present application, the feature construction module 2552 is also used to extract features from the first video frame features to obtain first features to be reconstructed; reconstruct the first features to be reconstructed by increasing the dimension to obtain first reconstructed features; fuse the first fused features based on the first video frame features and the first reconstructed features; and fuse the feature components of each channel in the first fused features to obtain the first mask area representing the position of the local information in the first video frame.

[0258] In some embodiments of the present application, the first fusion feature includes: a first fusion feature map, and the feature components include: a sub-fusion feature map; the feature construction module 2552 is further used to generate a channel feature vector for each channel based on the sub-fusion feature map of each channel of the first fusion feature map; calculate the corresponding attention weight for the channel feature vector of each channel; based on the attention weight, perform weighted fusion on the channel feature vector of each channel to obtain a fusion feature vector; restore the feature map for the fusion feature vector to obtain the first mask area representing the position of the local information in the first video frame.

[0259] In some embodiments of the present application, the feature extraction module 2551 is further used to divide the first video frame and the second video frame into blocks respectively to obtain multiple first image blocks and multiple second image blocks; perform convolution processing on the multiple first image blocks respectively to obtain multiple first convolution feature maps corresponding to the multiple first image blocks; perform convolution processing on the multiple second image blocks respectively to obtain multiple second convolution feature maps corresponding to the multiple second image blocks; determine the multiple first convolution feature maps as the first video frame features, and determine the multiple second convolution feature maps as the second video frame features.

[0260] The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device (i.e., the key frame detection device described above) reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the key frame detection method described in the present invention.

[0261] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the key frame detection method provided by the embodiment of the present application, for example, Figure 7 The key frame detection method is shown.

[0262] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0263] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0264] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0265] As an example, the executable instructions may be deployed to be executed on one computing device (i.e., the key frame detection device described above), or on multiple computing devices located in one location, or on multiple computing devices distributed in multiple locations and interconnected by a communication network.

[0266] To sum up, through the embodiments of the present application, the key frame detection device will simultaneously determine the temporal features of the position information representing the local information between the first video frame and the second video frame, as well as the spatial features representing the appearance of the local information in the first video frame and the second video frame, and further combine the features in the two dimensions of position information and appearance to derive the information similarity, so that the features considered when determining the similarity are more comprehensive, the accuracy of the judgment of information similarity is improved, and ultimately the accuracy of key frame detection is improved; and by reconstructing the first feature to be reconstructed in an increased dimension, the local information in the video frame can be more effectively located, so that the position representation of the local information is more accurate, so as to further improve the accuracy of the similarity calculation.

[0267] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A key frame detection method, characterized in that: The key frame detection method comprises: Performing feature extraction on a first video frame and a second video frame of the video to be detected, respectively, to obtain a first video frame feature of the first video frame and a second video frame feature of the second video frame; Based on the first video frame features and the second video frame features, constructing a temporal feature and a spatial feature; wherein the temporal feature represents a position change of local information in the video to be detected between the first video frame and the second video frame, and the spatial feature represents an appearance of the local information in the first video frame and the second video frame; Determining information similarity between the first video frame and the second video frame using the temporal features and the spatial features; According to the information similarity, a key frame corresponding to the local information in the video to be detected is determined.

2. The method according to claim 1, characterized in that The constructing of temporal features and spatial features based on the first video frame features and the second video frame features includes: Determining, based on the first video frame feature, a first mask region representing a position of the local information in the first video frame, and determining, based on the second video frame feature, a second mask region representing a position of the local information in the second video frame; fusing the first mask region and the first video frame feature into a first input feature corresponding to the first video frame, and fusing the second mask region and the second video frame feature into a second input feature corresponding to the second video frame; The temporal feature and the spatial feature of the local information are determined based on the first input feature and the second input feature.

3. The method according to claim 2, characterized in that The determining the temporal feature and the spatial feature of the local information based on the first input feature and the second input feature includes: Subtracting the first input feature from the second input feature to obtain a difference feature; The time series feature is obtained by performing dimensionality reduction on the difference feature; Performing dimensionality reduction on the first input feature to obtain a first reduced dimensionality feature, and performing dimensionality reduction on the second input feature to obtain a second reduced dimensionality feature; Performing differential processing on the first dimensionality reduction feature and the second dimensionality reduction feature to obtain the spatial feature.

4. The method according to claim 3, characterized in that The difference feature includes: a difference feature graph; the time series feature is obtained by performing dimensionality reduction on the difference feature, including: Performing a pooling operation on the difference feature map in a preset direction to obtain a pooling feature map; Splitting the pooled feature map according to the preset direction to obtain N sub-feature maps; wherein N is the length of the pooled feature map in the preset direction, and N is greater than or equal to 1; Performing pooling operations on the N sub-feature maps respectively to obtain pooling features corresponding to the N sub-feature maps; The pooled features corresponding to the N sub-feature maps are used to splice them into the temporal features.

5. The method according to any one of claims 1 to 4, characterized in that The information similarity includes text similarity; and determining the information similarity between the first video frame and the second video frame using the temporal feature and the spatial feature includes: Fusing the temporal features and the spatial features to obtain spatiotemporal fusion features; The similarity of the spatiotemporal fusion features is performed using a similarity judgment model to obtain text similarity between the first video frame and the second video frame.

6. The method according to claim 5, characterized in that Determining, based on the information similarity, a key frame corresponding to the local information in the video to be detected includes: When the text similarity is less than a similarity threshold, determining both the first video frame and the second video frame as the key frames corresponding to the local information in the video to be detected; When the text similarity is greater than or equal to the similarity threshold, the first video frame or the second video frame is determined as the key frame corresponding to the local information in the video to be detected.

7. The method according to claim 6, characterized in that The method further comprises: When the text similarity is greater than or equal to the similarity threshold, fusing the first video frame and the second video frame to obtain a fused video frame; The fused video frame is determined as the key frame corresponding to the local information in the video to be detected.

8. The method according to claim 2, characterized in that The determining, based on the first video frame feature, a first mask region representing a position of the local information in the first video frame includes: Performing feature extraction on the first video frame feature to obtain a first feature to be reconstructed; Reconstructing the first feature to be reconstructed by increasing the dimension to obtain a first reconstructed feature; fusing a first fused feature based on the first video frame feature and the first reconstructed feature; The feature components of each channel in the first fusion feature are fused to obtain the first mask area representing the position of the local information in the first video frame.

9. The method according to claim 8, characterized in that The first fusion feature includes: a first fusion feature map, and the feature components include: a sub-fusion feature map; the feature components of each channel in the first fusion feature are fused to obtain the first mask area representing the position of the local information in the first video frame, including: Generating a channel feature vector for each channel according to the sub-fused feature map of each channel of the first fused feature map; Calculating a corresponding attention weight for the channel feature vector of each channel; Based on the attention weight, performing weighted fusion on the channel feature vectors of each channel to obtain a fused feature vector; A feature map is restored for the fused feature vector to obtain the first mask region representing the position of the local information in the first video frame.

10. The method according to any one of claims 1 to 4, 8 or 9, characterized in that The extracting features of the first video frame and the second video frame of the video to be detected respectively to obtain first video frame features of the first video frame and second video frame features of the second video frame includes: Divide the first video frame and the second video frame into blocks respectively to obtain a plurality of first image blocks and a plurality of second image blocks; Performing convolution processing on each of the first image blocks to obtain a plurality of first convolution feature maps corresponding to the first image blocks; Performing convolution processing on each of the plurality of second image blocks to obtain a plurality of second convolution feature maps corresponding to the plurality of second image blocks; The multiple first convolution feature maps are determined as the first video frame features, and the multiple second convolution feature maps are determined as the second video frame features.

11. A key frame detection device, characterized in that: The key frame detection device comprises: a feature extraction module, configured to extract features from a first video frame and a second video frame of the video to be detected, respectively, to obtain first video frame features of the first video frame and second video frame features of the second video frame; a feature construction module, configured to construct a temporal feature and a spatial feature based on the first video frame feature and the second video frame feature; wherein the temporal feature represents a position change of local information in the video to be detected between the first video frame and the second video frame, and the spatial feature represents an appearance of the local information in the first video frame and the second video frame; a similarity determination module, configured to determine information similarity between the first video frame and the second video frame using the temporal features and the spatial features; The key frame determination module is used to determine the key frame corresponding to the local information in the video to be detected based on the information similarity.

12. A key frame detection device, characterized in that: The key frame detection device comprises: a memory for storing executable instructions; The processor is configured to implement the key frame detection method according to any one of claims 1 to 10 when executing the executable instructions stored in the memory.

13. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the key frame detection method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the key frame detection method according to any one of claims 1 to 10 is implemented.

Images