Video frame extraction methods and apparatus, electronic devices and storage media

Abstract

This application provides a video frame extraction method, apparatus, electronic device, and storage medium, belonging to the field of artificial intelligence technology and applicable to the fintech and healthcare fields. The method includes: extracting sample image feature sequences from sample video frames of a sample video; evaluating a first uncertainty score for each sample video frame using an initial evaluation model; extracting at least two neighboring video frames from the sample video and calculating the feature similarity between the sample video frame and each neighboring video frame; calculating pseudo-label weights based on the first uncertainty score and feature similarity, and constructing loss data based on the pseudo-label weights and feature similarity; adjusting the parameters of the initial evaluation model based on the loss data to obtain a target evaluation model; and evaluating a second uncertainty score for the target video frame using the target evaluation model to extract key video frames. This application embodiment can accurately extract key frames from videos.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of fintech and healthcare, and particularly to a video frame extraction method and apparatus, electronic device and storage medium. Background Technology

[0002] Currently, keyframes in videos can be applied to video analysis tasks such as video summarization, video indexing, and video compression. In the medical field, keyframes representing each important step (such as incision, suturing, and hemostasis) can be extracted from surgical videos, and video indexes can be generated based on these keyframes to facilitate video backtracking. In the fintech field, keyframes can be extracted from videos recording transaction behavior so that important actions (such as withdrawals and signatures) can be quickly located during subsequent backtracking.

[0003] In related technologies, the similarity between any two adjacent video frames is calculated, and the pseudo-label of the current frame is inferred to be either a keyframe or a non-keyframe based on this similarity. The model is then trained based on these pseudo-labels. However, this method is unsuitable for videos with fast motion or scene changes, resulting in low accuracy in keyframe extraction. Therefore, accurately extracting keyframes from videos has become a pressing technical problem. Summary of the Invention

[0004] The main objective of this application is to provide a video frame extraction method, apparatus, electronic device, and storage medium, which aims to improve the accuracy of video keyframes.

[0005] To achieve the above objectives, a first aspect of this application proposes a video frame extraction method, the method comprising:

[0006] A sample video is acquired, and image features are extracted from the sample video frames of the sample video to obtain a sample image feature sequence; wherein, the sample video frames have sample timestamps; The sample image feature sequence is classified and uncertain by using a preset initial evaluation model to obtain the first uncertainty score for each sample video frame; For the sample video frame, at least two neighboring video frames are extracted from the sample video, and the feature similarity between the sample video frame and each of the neighboring video frames is calculated; wherein, each of the neighboring video frames has a neighboring timestamp, and at least one of the neighboring timestamps is the timestamp before or after the timestamp of the sample timestamp. The pseudo-label weight is calculated based on the first uncertainty score and the feature similarity. Loss data is constructed based on the pseudo-label weights and the feature similarity; Based on the loss data, the parameters of the initial evaluation model are adjusted to obtain the target evaluation model; Acquire a target video, and use the target evaluation model to perform classification uncertainty evaluation on the target video frames of the target video to obtain a second uncertainty score for each target video frame; Based on the second uncertainty score, key video frames are selected from the target video frames, and the key video frames are extracted from the target video.

[0007] In some embodiments, calculating the pseudo-label weight based on the first uncertainty score and the feature similarity includes: Based on the sample timestamp, consecutive historical video frames are selected from the sample video, and the first uncertainty score of the historical video frames is used as the reference uncertainty score; wherein, the historical video frames have historical timestamps, and the historical timestamps are earlier than the sample timestamps; The pseudo-label weights are obtained by calculating the weights based on the first uncertainty score and the reference uncertainty score of the sample video frames.

[0008] In some embodiments, the step of calculating the pseudo-label weights based on the first uncertainty score and the reference uncertainty score of the sample video frames includes: Attention encoding is performed on the first uncertainty score and the reference uncertainty score of the sample video frame to obtain a score feature vector; The sensitivity parameter is obtained by mapping the score feature vector according to a preset activation function; The pseudo-label weight is obtained by performing an exponential calculation based on the sensitivity parameter and the first uncertainty score.

[0009] In some embodiments, mapping the score feature vector according to a preset activation function to obtain a sensitivity parameter includes: Obtain the bias vector and the transpose of the model weights; Based on the bias vector, the model weight transpose matrix and the score feature vector are weighted and calculated to obtain the candidate vector; The sensitivity parameter is obtained based on the activation function and the candidate vector.

[0010] In some embodiments, constructing loss data based on the pseudo-label weights and the feature similarity includes: For each of the sample video frames, pseudo-label data is determined based on the minimum value of the feature similarity; Loss calculation is performed based on the pseudo-label data and the pseudo-label weights to obtain frame-level loss sub-data; The loss data is obtained by weighted averaging of the frame-level loss sub-data of all the sample video frames.

[0011] In some embodiments, selecting key video frames from the target video frames based on the second uncertainty score includes: Sort the second uncertainty scores in descending order to obtain the score sequence; A preset number of intermediate scores are selected based on the position identifiers of the second uncertainty score in the score sequence; wherein the position identifiers of the intermediate scores are less than or equal to the preset number. Key video frames are determined from the target video frames based on the intermediate scores.

[0012] In some embodiments, the step of evaluating the classification uncertainty of the sample image feature sequence using a preset initial evaluation model to obtain a first uncertainty score for each sample video frame includes: Temporal feature encoding is performed on the feature sequence of the sample image to obtain a temporal feature sequence; The initial evaluation model is used to predict the first uncertainty score for each feature in the time-series feature sequence.

[0013] To achieve the above objectives, a second aspect of this application provides a video frame extraction apparatus, the apparatus comprising: The acquisition module is used to acquire sample videos and extract image features from sample video frames to obtain sample image feature sequences; wherein, the sample video frames have sample timestamps; The first evaluation module is used to evaluate the classification uncertainty of the sample image feature sequence through a preset initial evaluation model, and obtain the first uncertainty score for each sample video frame. The similarity calculation module is used to extract at least two neighboring video frames from a sample video frame and calculate the feature similarity between the sample video frame and each neighboring video frame; wherein each neighboring video frame has a neighboring timestamp, and at least one neighboring timestamp is the timestamp before or after the sample timestamp. The weight calculation module is used to calculate the pseudo-label weights based on the first uncertainty score and feature similarity. The loss data construction module is used to construct loss data based on pseudo-label weights and feature similarity. The parameter adjustment module is used to adjust the parameters of the initial evaluation model based on the loss data to obtain the target evaluation model. The second evaluation module is used to acquire the target video and perform classification uncertainty evaluation on the target video frames of the target video using the target evaluation model to obtain the second uncertainty score for each target video frame. The video frame extraction module is used to select key video frames from the target video frames based on the second uncertainty score, and to extract key video frames from the target video.

[0014] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0016] The video frame extraction method, apparatus, electronic device, and storage medium proposed in this application extract image features from sample videos to obtain sample image feature sequences. Then, the classification uncertainty of the sample image feature sequences is evaluated to obtain a first uncertainty score for each sample video frame. Next, at least two neighboring video frames are selected from each sample video frame, and the feature similarity between them is calculated. Then, pseudo-label weights are calculated based on the first uncertainty score and feature similarity, enabling the model to dynamically assign corresponding weights to video frames with high uncertainty or rapid dynamic changes, thereby reducing the impact of noise. Then, loss data is constructed based on the pseudo-label weights and feature similarity. Subsequently, the parameters of the initial evaluation model are adjusted based on the loss data, and keyframes are selected from the target video frames using the adjusted target evaluation model. Finally, key video frames are extracted from the target video. Compared to existing technologies, the embodiments of this application, by performing uncertainty evaluation and calculating pseudo-label weights for each video frame, and training the initial evaluation model based on the loss data constructed from the pseudo-label weights and feature similarity, can effectively reduce pseudo-label noise caused by rapid motion or scene switching, thereby improving the accuracy of keyframe extraction. Attached Figure Description

[0017] Figure 1 This is a flowchart of the video frame extraction method provided in the embodiments of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the document; Figure 3 yes Figure 1 The flowchart of step S105 in the process; Figure 4 yes Figure 3 The flowchart of step S302 in the document; Figure 5 This is a schematic diagram of the video frame extraction device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0022] The Dirichlet distribution is a continuous multivariate distribution suitable for determining the probabilities of events in a multinomial distribution. It is defined in a multidimensional probability simplex, a vector space where the sum of all elements is 1. For example, considering a case with K classes, the Dirichlet distribution can model K probability values ​​(p1, p2, ..., pK), representing the probability of each class.

[0023] Currently, keyframes in videos can be applied to video analysis tasks such as video summarization, video indexing, and video compression. In the medical field, keyframes representing each important step (such as incision, suturing, and hemostasis) can be extracted from surgical videos, and video indexes can be generated based on these keyframes to facilitate video backtracking. In the fintech field, keyframes can be extracted from videos recording transaction behavior so that important actions (such as withdrawals and signatures) can be quickly located during subsequent backtracking.

[0024] In related technologies, the similarity between any two adjacent video frames is calculated, and the pseudo-label of the current frame is inferred to be either a keyframe or a non-keyframe based on this similarity. The model is then trained based on these pseudo-labels. However, this method is unsuitable for videos with fast motion or scene changes, resulting in low accuracy in keyframe extraction. Therefore, accurately extracting keyframes from videos has become a pressing technical problem.

[0025] Based on this, embodiments of this application provide a video frame extraction method and apparatus, electronic device and storage medium, aiming to improve the accuracy of video keyframes.

[0026] The video frame extraction method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the video frame extraction method in this application is described.

[0027] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0028] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0029] The video frame extraction method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the video frame extraction method, but is not limited to the above forms.

[0030] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0031] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0032] Figure 1 This is an optional flowchart of the video frame extraction method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S108.

[0033] Step S101: Obtain the sample video and extract image features from the sample video frames of the sample video to obtain the sample image feature sequence; wherein, the sample video frames have sample timestamps.

[0034] Step S102: The classification uncertainty of the sample image feature sequence is evaluated by a preset initial evaluation model to obtain the first uncertainty score of each sample video frame.

[0035] Step S103: For the sample video frame, extract at least two neighboring video frames from the sample video and calculate the feature similarity between the sample video frame and each neighboring video frame; wherein, each neighboring video frame has a neighboring timestamp, and at least one neighboring timestamp is the timestamp before or after the sample timestamp.

[0036] Step S104: Calculate the pseudo-label weights based on the first uncertainty score and feature similarity.

[0037] Step S105: Construct loss data based on pseudo-label weights and feature similarity.

[0038] Step S106: Adjust the parameters of the initial evaluation model based on the loss data to obtain the target evaluation model.

[0039] Step S107: Obtain the target video, and use the target evaluation model to evaluate the classification uncertainty of the target video frames to obtain the second uncertainty score for each target video frame.

[0040] Step S108: Select key video frames from the target video frames based on the second uncertainty score, and extract key video frames from the target video.

[0041] Steps S101 to S108 of this embodiment involve extracting image features from the sample video to obtain a sample image feature sequence. Then, a classification uncertainty assessment is performed on the sample image feature sequence to obtain a first uncertainty score for each sample video frame. Next, at least two neighboring video frames are selected from each sample video frame, and the feature similarity between them is calculated. Then, pseudo-label weights are calculated based on the first uncertainty score and feature similarity, enabling the model to dynamically assign corresponding weights to video frames with high uncertainty or rapid dynamic changes, thereby reducing the impact of noise. Then, loss data is constructed based on the pseudo-label weights and feature similarity. Subsequently, the parameters of the initial evaluation model are adjusted based on the loss data, and keyframes are selected for the target video frame using the adjusted target evaluation model. Finally, keyframes are extracted from the target video. Compared to existing technologies, this embodiment, by performing uncertainty assessment and calculating pseudo-label weights for each video frame, and training the initial evaluation model based on the loss data constructed from the pseudo-label weights and feature similarity, can effectively reduce pseudo-label noise caused by rapid motion or scene switching, thereby improving the accuracy of keyframe extraction.

[0042] In step S101 of some embodiments, the sample video is a video data source used for training, and its video format can be AVI, MP4, MKV, etc. A sample video frame is a single static image in the sample video, that is, each frame image in the sample video sequence, and the sample timestamp is a timestamp of each frame image, indicating the specific time position of the frame image in the sample video.

[0043] Understandably, deep neural networks (e.g., convolutional neural networks (CNNs)) can be used to extract features from each frame of the image. Commonly used network architectures include ResNet, VGG, and Inception. The image features of all sample video frames are arranged according to the sample timestamps to obtain a sequence of sample image features.

[0044] In step S102 of some embodiments, the initial evaluation model is a model used to evaluate the uncertainty of video frame classification. A first uncertainty score is a metric used to quantify the uncertainty of image classification prediction for each frame, aiming to assess the model's confidence in predicting whether each frame is a keyframe. A higher uncertainty score indicates uncertainty in the predicted category of a frame; conversely, a lower uncertainty score indicates greater certainty in the predicted category.

[0045] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S202: Step S201: Perform temporal feature encoding on the sample image feature sequence to obtain a temporal feature sequence.

[0046] Step S202: Predict the first uncertainty score for each feature in the time-series feature sequence using the initial evaluation model.

[0047] In step S201 of some embodiments, the sample image feature sequence can be encoded by models such as Long Short-Term Memory Network (LSTM), Quasi-Recurrent LSTM, Recurrent Neural Network (RNN) or Gated Recurrent Unit (GRU) to obtain a temporal representation containing temporal context information, namely the temporal feature sequence.

[0048] In step S202 of some embodiments, the input temporal features are first processed through a fully connected layer (FC) to predict the parameters of the Dirichlet distribution. ,in It is the number of pseudo-categories predefined by clustering. The calculation process for the first uncertainty score can be referenced in the following analytical formula: (1), in, It is the number of pseudo-categories predefined by clustering. This represents the confidence level of the initial evaluation model for class k at time t. Let represent the first uncertainty score of the t-th sample video frame. This represents the total amount of evidence, used to quantify the overall confidence level across all categories at time t. A large amount of evidence indicates that the model is highly certain about its predictions. It approaches 0. Conversely, when the amount of evidence is small (i.e., the model prediction is uncertain), it approaches 0. Approaching 1.

[0049] Steps S201 to S202, as illustrated in the embodiments of this application, enable the initial evaluation model to capture the temporal dependencies between video frames through temporal feature encoding, significantly improving the ability to understand the dynamic changes of video frames during fast motion and scene transitions. The prediction of the first uncertainty score further quantifies the credibility of each frame, providing a stronger basis for the accurate identification of keyframes.

[0050] In step S103 of some embodiments, the neighboring video frames are video frames that are a certain number of frames adjacent to a given frame in the video. For example, in a sample video, if the current frame is frame t, then its temporal neighboring frames could be frame-1, t+1, t-2, t+2, etc. In this embodiment, the cosine similarity between the sample video frame and each neighboring video frame is calculated as the feature similarity.

[0051] In step S104 of some embodiments, the pseudo-label weight is a weight value calculated based on the uncertainty score and feature similarity, used to adjust the label credibility of the sample video frame.

[0052] Please see Figure 3 In some embodiments, step S104 may include, but is not limited to, steps S301 to S302: Step S301: Select consecutive historical video frames from the sample video according to the sample timestamp, and use the first uncertainty score of the historical video frame as the reference uncertainty score.

[0053] Step S302: Calculate the weights based on the first uncertainty score and the reference uncertainty score of the sample video frame to obtain the pseudo-label weights.

[0054] In step S301 of some embodiments, the historical video frames have historical timestamps that are earlier than the sample timestamps. Historical video frames refer to those frames on the video timeline that are earlier than the current sample video frame. These historical frames are used to provide temporal context information. The reference uncertainty score refers to the first uncertainty score of the historical video frame. The number of historical video frames can be 10; this embodiment does not strictly limit its specific value.

[0055] In step S302 of some embodiments, the weight calculation process includes: 1) performing attention encoding based on the first uncertainty score and the reference uncertainty score of the sample video frame to obtain a score feature vector. Specifically, self-attention calculation can be performed on the first uncertainty score and the reference uncertainty score to obtain the score feature vector.

[0056] 2) Map the score feature vector according to the preset activation function to obtain the sensitivity parameter. The sensitivity parameter is used to control the weighting intensity. In this embodiment, if the video reflects a more static scene, a smaller sensitivity parameter is needed. If the video reflects a more dynamic scene, a larger sensitivity parameter is needed. Further, the process of obtaining the sensitivity parameter includes: 2.1) Obtain the bias vector and the transpose of the model weights; the transpose of the model weights is the transpose of the learnable weights matrix. 2.2) Perform a weighted calculation based on the bias vector, the transpose of the model weights matrix, and the score feature vector to obtain the candidate vector. 2.3) Obtain the sensitivity parameter based on the activation function and the candidate vector. In this embodiment, the activation function can be the Sigmoid function. The above calculation process can be expressed analytically as follows: (2), in, This represents the sensitivity parameter corresponding to the t-th sample video frame. m represents the number of historical video frames. 'b' represents the candidate vector. 'b' represents the bias vector. This represents the transpose of the model weights. () indicates a self-attention computation operation. () represents the activation function.

[0057] 3) Calculate the pseudo-label weights by performing an index calculation based on the sensitivity parameter and the first uncertainty score. The calculation of the pseudo-label weights can be expressed by the following analytical formula: (3), in, This indicates the weight of the pseudo-label. This represents the sensitivity parameter corresponding to the t-th sample video frame. Let represent the first uncertainty score corresponding to the t-th sample video frame.

[0058] Steps S301 to S302 of this embodiment propose a technical solution to improve the accuracy of keyframe extraction by combining the feature similarity of video frames, timestamp information, and uncertainty scores. By determining pseudo-label data based on the minimum value in the feature similarity, the method of this embodiment can effectively eliminate noise caused by inaccurate similarity between video frames, thereby enhancing the reliability of pseudo-labels. Furthermore, weight calculation is performed based on the first uncertainty score of the sample video frame and the reference uncertainty score of historical video frames, enabling the model to adjust the weight of each frame more reasonably according to the timestamp information, thereby improving the accuracy of keyframe extraction.

[0059] In step S105 of some embodiments, the loss data is a loss function used to train the model.

[0060] Please see Figure 4 In some embodiments, step S105 may include, but is not limited to, steps S401 to S403: Step S401: For each sample video frame, determine the pseudo-label data based on the minimum value of the feature similarity.

[0061] Step S402: Calculate the loss based on the pseudo-label data and pseudo-label weights to obtain frame-level loss sub-data.

[0062] Step S403: Take a weighted average of the frame-level loss sub-data of all sample video frames to obtain the loss data.

[0063] In step S401 of some embodiments, the pseudo-tag data can be represented by the following analytical expression: (4), in, This represents the pseudo-label data for the current t-th frame. ( ) indicates the calculation of cosine similarity. Let represent the image features of the sample video frame at frame t. Represents the image features of neighboring video frames. This represents the temporal neighborhood of frame t. arg min represents the operation of finding the minimum value.

[0064] In step S402 of some embodiments, frame-level loss sub-data refers to the loss data calculated separately for each frame of video during the loss calculation process, reflecting the contribution of each frame to the overall loss. Frame-level loss sub-data can be represented as... ,in This represents pseudo-label data. It represents the multiplication operation. This indicates the feature similarity to the pseudo-labeled data. This represents the pseudo-label data for the current t-th frame.

[0065] In step S403 of some embodiments, the calculation process of the weighted average can refer to the following analytical expression: (5), in, express. This represents the frame-level loss sub-data, and T represents the total number of sample video frames.

[0066] Steps S401 to S403 as shown in the embodiments of this application can obtain more stable and reliable loss data, improve the performance of the evaluation model, and ultimately ensure the accuracy of keyframe extraction.

[0067] In step S106 of some embodiments, the target evaluation model is an initial evaluation model optimized based on the loss data. Parameter adjustment can be stopped when the loss data converges, or the first uncertainty score approaches 0, or the number of training rounds reaches a preset number (which can be 10).

[0068] In step S107 of some embodiments, the target video is the video data to be evaluated. The second uncertainty score is the confidence level of each frame obtained after performing classification uncertainty evaluation on the target video frames. The implementation of classification uncertainty evaluation is the same as in the embodiment of step S102, and will not be described again here.

[0069] In step S108 of some embodiments, the process of selecting key video frames may include: 1) sorting the second uncertainty scores in descending order to obtain a score sequence. For example, assuming the target video contains 10 frames, the second uncertainty scores are as follows: [0.85, 0.70, 0.95, 0.60, 0.80, 0.65, 0.90, 0.75, 0.88, 0.82]. After sorting in descending order, the obtained score sequence is as follows: [0.95, 0.90, 0.88, 0.85, 0.82, 0.80, 0.75, 0.70, 0.65, 0.60].

[0070] 2) Select a preset number of intermediate scores based on the position identifier of the second uncertainty score in the score sequence. The position identifier of each intermediate score is less than or equal to the preset number. For example, assuming the preset number is 4, the first 4 intermediate scores are selected based on the position identifier of the second uncertainty score in the score sequence, i.e., [0.95, 0.90, 0.88, 0.85]. These scores will be used as intermediate scores.

[0071] 3) Identify key video frames from the target video frames based on the intermediate scores. Continuing with the example from the previous step, the frames corresponding to the intermediate scores are frames 3, 7, 9, and 1. These four frames will be identified as key video frames. Finally, these key video frames are extracted from the target video.

[0072] Please see Figure 5 This application also provides a video frame extraction apparatus that can implement the above-described video frame extraction method. The apparatus includes: The acquisition module 501 is used to acquire sample videos and extract image features from sample video frames of the sample videos to obtain sample image feature sequences; wherein, the sample video frames have sample timestamps.

[0073] The first evaluation module 502 is used to evaluate the classification uncertainty of the sample image feature sequence through a preset initial evaluation model, and obtain the first uncertainty score of each sample video frame.

[0074] The similarity calculation module 503 is used to extract at least two neighboring video frames from the sample video for a sample video frame, and calculate the feature similarity between the sample video frame and each neighboring video frame; wherein each neighboring video frame has a neighboring timestamp, and at least one neighboring timestamp is the timestamp before or after the sample timestamp.

[0075] The weight calculation module 504 is used to calculate the pseudo-label weight based on the first uncertainty score and feature similarity.

[0076] The loss data construction module 505 is used to construct loss data based on pseudo-label weights and feature similarity.

[0077] The parameter adjustment module 506 is used to adjust the parameters of the initial evaluation model based on the loss data to obtain the target evaluation model.

[0078] The second evaluation module 507 is used to acquire the target video, and to evaluate the classification uncertainty of the target video frames through the target evaluation model to obtain the second uncertainty score of each target video frame.

[0079] The video frame extraction module 508 is used to select key video frames from the target video frames based on the second uncertainty score, and to extract key video frames from the target video.

[0080] The specific implementation of this video frame extraction device is basically the same as the specific embodiment of the video frame extraction method described above, and will not be repeated here.

[0081] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described video frame extraction method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0082] Please see Figure 6 , Figure 6 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 602 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601 using the video frame extraction method of the embodiments of this application. The input / output interface 603 is used to implement information input and output; The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604); The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0083] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video frame extraction method.

[0084] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0085] The video frame extraction method, device, electronic device, and storage medium provided in this application extract image features from sample videos to obtain sample image feature sequences. Then, the classification uncertainty of the sample image feature sequences is evaluated to obtain a first uncertainty score for each sample video frame. Next, at least two neighboring video frames are selected from each sample video frame, and the feature similarity between them is calculated. Then, pseudo-label weights are calculated based on the first uncertainty score and feature similarity, enabling the model to dynamically assign corresponding weights to video frames with high uncertainty or rapid dynamic changes, thereby reducing the impact of noise. Then, loss data is constructed based on the pseudo-label weights and feature similarity. Subsequently, the parameters of the initial evaluation model are adjusted based on the loss data, and keyframes are selected for the target video frame using the adjusted target evaluation model. Finally, key video frames are extracted from the target video. Compared to existing technologies, this application, by evaluating uncertainty and calculating pseudo-label weights for each video frame, and training the initial evaluation model based on the loss data constructed from the pseudo-label weights and feature similarity, can effectively reduce pseudo-label noise caused by rapid motion or scene switching, thereby improving the accuracy of keyframe extraction.

[0086] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0087] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0090] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0091] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0093] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A video frame extraction method, characterized in that, The method includes: A sample video is acquired, and image features are extracted from the sample video frames of the sample video to obtain a sample image feature sequence; wherein, the sample video frames have sample timestamps; The sample image feature sequence is classified and uncertain by using a preset initial evaluation model to obtain the first uncertainty score for each sample video frame; For the sample video frame, at least two neighboring video frames are extracted from the sample video, and the feature similarity between the sample video frame and each of the neighboring video frames is calculated; wherein, each of the neighboring video frames has a neighboring timestamp, and at least one of the neighboring timestamps is the timestamp before or after the timestamp of the sample timestamp. The pseudo-label weight is calculated based on the first uncertainty score and the feature similarity. Loss data is constructed based on the pseudo-label weights and the feature similarity; Based on the loss data, the parameters of the initial evaluation model are adjusted to obtain the target evaluation model; Acquire a target video, and use the target evaluation model to perform classification uncertainty evaluation on the target video frames of the target video to obtain a second uncertainty score for each target video frame; Based on the second uncertainty score, key video frames are selected from the target video frames, and the key video frames are extracted from the target video.

2. The method according to claim 1, characterized in that, The step of calculating the pseudo-label weight based on the first uncertainty score and the feature similarity includes: Based on the sample timestamp, consecutive historical video frames are selected from the sample video, and the first uncertainty score of the historical video frames is used as the reference uncertainty score; wherein, the historical video frames have historical timestamps, and the historical timestamps are earlier than the sample timestamps; The pseudo-label weights are obtained by calculating the weights based on the first uncertainty score and the reference uncertainty score of the sample video frames.

3. The method according to claim 2, characterized in that, The step of calculating the pseudo-label weights based on the first uncertainty score and the reference uncertainty score of the sample video frames includes: Attention encoding is performed on the first uncertainty score and the reference uncertainty score of the sample video frame to obtain a score feature vector; The sensitivity parameter is obtained by mapping the score feature vector according to a preset activation function; The pseudo-label weight is obtained by performing an exponential calculation based on the sensitivity parameter and the first uncertainty score.

4. The method according to claim 3, characterized in that, The step of mapping the score feature vector according to a preset activation function to obtain the sensitivity parameter includes: Obtain the bias vector and the transpose of the model weights; Based on the bias vector, the model weight transpose matrix and the score feature vector are weighted and calculated to obtain the candidate vector; The sensitivity parameter is obtained based on the activation function and the candidate vector.

5. The method according to claim 1, characterized in that, The step of constructing loss data based on the pseudo-label weights and the feature similarity includes: For each of the sample video frames, pseudo-label data is determined based on the minimum value of the feature similarity; Loss calculation is performed based on the pseudo-label data and the pseudo-label weights to obtain frame-level loss sub-data; The loss data is obtained by weighted averaging of the frame-level loss sub-data of all the sample video frames.

6. The method according to any one of claims 1 to 5, characterized in that, The step of selecting key video frames from the target video frames based on the second uncertainty score includes: Sort the second uncertainty scores in descending order to obtain the score sequence; A preset number of intermediate scores are selected based on the position identifiers of the second uncertainty score in the score sequence; wherein the position identifiers of the intermediate scores are less than or equal to the preset number. Key video frames are determined from the target video frames based on the intermediate scores.

7. The method according to any one of claims 1 to 5, characterized in that, The step of evaluating the classification uncertainty of the sample image feature sequence using a preset initial evaluation model to obtain a first uncertainty score for each sample video frame includes: Temporal feature encoding is performed on the feature sequence of the sample image to obtain a temporal feature sequence; The initial evaluation model is used to predict the first uncertainty score for each feature in the time-series feature sequence.

8. A video frame extraction device, characterized in that, The device includes: The acquisition module is used to acquire sample videos and extract image features from sample video frames of the sample videos to obtain sample image feature sequences; wherein, the sample video frames have sample timestamps; The first evaluation module is used to evaluate the classification uncertainty of the sample image feature sequence through a preset initial evaluation model, and obtain the first uncertainty score of each sample video frame. The similarity calculation module is used to extract at least two neighboring video frames from the sample video frame and calculate the feature similarity between the sample video frame and each of the neighboring video frames; wherein each of the neighboring video frames has a neighboring timestamp, and at least one of the neighboring timestamps is the timestamp before or after the timestamp of the sample timestamp. The weight calculation module is used to calculate the pseudo-label weight based on the first uncertainty score and the feature similarity. A loss data construction module is used to construct loss data based on the pseudo-label weights and the feature similarity. The parameter adjustment module is used to adjust the parameters of the initial evaluation model based on the loss data to obtain the target evaluation model. The second evaluation module is used to acquire the target video, perform classification uncertainty evaluation on the target video frames of the target video using the target evaluation model, and obtain a second uncertainty score for each target video frame. The video frame extraction module is used to select key video frames from the target video frames based on the second uncertainty score, and to extract the key video frames from the target video.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images