Sampling method and device of video frame, nonvolatile storage medium and electronic device

Abstract

The application discloses a video frame sampling method and device, a nonvolatile storage medium and an electronic device. The method comprises the following steps: receiving a video to be sampled and a sampling requirement, wherein the sampling requirement records a preset sampling quantity; determining a first correlation index of each video frame contained in the video to be sampled; screening all the video frames contained in the video to be sampled according to the first correlation index of each video frame and the preset sampling quantity, and obtaining an initial video frame set; iteratively performing a dynamic updating strategy on the video frame set to be updated until an iteration stopping condition is reached, and outputting a target video frame set. The application solves the technical problems of rigid sampling strategy, low sampling efficiency and low accuracy of sampling results caused by the single factor of considering only the correlation of the video frame in the key frame sampling method for the video in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a video frame sampling method and device, a non-volatile storage medium and an electronic device. BACKGROUND

[0002] With the rapid development of digital video technology, the amount of video data is growing explosively. In application scenarios such as video summary, video retrieval and video monitoring, key frame extraction technology is crucial. Key frames are video frames that are highly related to the theme of the video and can cover different contents of the video, and can reflect the diversity of the video to which they belong. In related technologies, when sampling and extracting key frames from a video, only the correlation between the video frames and the video theme is considered, the purpose is to ensure that the extracted key frames can accurately reflect the core content of the video. Using this method, the video content of the extracted key frames is single, there is a problem that different scenes and details in the video cannot be fully covered, and only according to the correlation between the video frames and the video theme for sampling without considering other factors, there is also a problem that the sampling strategy is rigid.

[0003] At present, no effective solution has been proposed for the above problems. SUMMARY

[0004] The embodiments of the present application provide a video frame sampling method and device, a non-volatile storage medium and an electronic device, to at least solve the technical problems of rigid sampling strategy, low sampling efficiency and low accuracy of sampling results caused by the fact that the method of sampling key frames from a video in related technologies only considers the correlation of video frames.

[0005] According to an aspect of the embodiments of the present application, a method for sampling video frames is provided, comprising: receiving a video to be sampled and a sampling requirement, wherein the sampling requirement records a preset sampling number, the preset sampling number is used to indicate the number of video frames extracted from the video to be sampled; for each video frame contained in the video to be sampled, determining a first correlation index of each video frame, and screening all video frames contained in the video to be sampled according to the first correlation index of each video frame and the preset sampling number to obtain an initial video frame set, wherein the first correlation index is used to quantitatively evaluate the correlation between the video frame and the video theme of the video to be sampled; iteratively performing a dynamic updating strategy on the updated video frame set until an iteration stopping condition is reached, and outputting a target video frame set, wherein the target video frame set is the video frame set output by the last execution of the dynamic updating strategy; wherein the dynamic updating strategy comprises: determining a first score of each candidate video frame contained in a candidate frame set, wherein the first score is used to quantitatively evaluate the correlation degree between the candidate video frame and the video theme, and the candidate frame set is a set composed of remaining video frames, the remaining video frames being the video frames in the video to be sampled that are not contained in the updated video frame set; updating the updated video frame set according to the first score, the preset sampling number and the candidate frame set to obtain a new video frame set; determining a plurality of content evaluation indexes of the new video frame set; determining an update result of a sampling parameter according to the plurality of content evaluation indexes, wherein the sampling parameter is used to guide the updating process of the updated video frame set; outputting the new video frame set and the update result of the sampling parameter.

[0006] Optionally, determining the first correlation index of each video frame comprises: determining a video theme of the video to be sampled, and encoding the video theme into a vector form to obtain a video theme embedding vector; and performing feature extraction on each video frame to obtain a feature vector of each video frame, wherein the feature vector records multi-dimensional information of the video frame, and the multi-dimensional information comprises visual information and text information of an image recorded by the video frame; for each video frame, determining a similarity between the feature vector and the video theme embedding vector, and determining the similarity as the first correlation index of the video frame.

[0007] Optionally, screening all video frames contained in the video to be sampled according to the first correlation index of each video frame and the preset sampling number to obtain the initial video frame set comprises: determining a second score of each video frame according to an initialization parameter and the first correlation index of each video frame, wherein each second score is used to quantitatively evaluate the correlation degree between each video frame and the video theme; sorting all video frames according to the second score of each video frame to obtain a video frame sequence, and screening the video frame sequence according to the arrangement order of the video frames in the video frame sequence and the preset sampling number to obtain the initial video frame set.

[0008] Optionally, the initialization parameters comprise: a preset initial dissimilarity and a preset information gain ratio of each video frame; and the second score of each video frame is determined according to the initialization parameters and the first correlation index of each video frame, comprising: for each video frame, determining a first product of a first weight corresponding to the dissimilarity and the preset initial dissimilarity, a second product of a second weight corresponding to the correlation index and the first correlation index, and a third product of a third weight corresponding to the information gain ratio and the preset information gain ratio, and determining a sum of the first product, the second product and the third product as the second score.

[0009] Optionally, the first score of each candidate video frame contained in the candidate frame set is determined, comprising: obtaining historical sampling parameters carried by the candidate video frame corresponding to the to-be-updated video frame set, wherein the historical sampling parameters are contained in the output of the dynamic updating strategy to which the to-be-updated video frame set belongs; determining a second correlation index of each candidate video frame and a dissimilarity between the candidate video frame and a historical video frame set, wherein the historical video set is a video frame set output by the last execution of the dynamic updating strategy; and determining the first score of each candidate video frame according to the historical sampling parameters, the second correlation index and the dissimilarity.

[0010] Optionally, the to-be-updated video frame set is updated according to the first score, the preset sampling number and the candidate frame set to obtain a new video frame set, comprising: sorting all candidate video frames contained in the candidate frame set according to the first score corresponding to each candidate video frame to obtain a candidate video frame sequence; and screening the candidate frame set according to the sorting of the candidate video frame in the candidate video frame sequence and the preset sampling number to obtain the new video frame set, wherein the new video frame set is an update result of the to-be-updated video frame set, the to-be-updated video frame set is an initial video frame set when the dynamic updating strategy is executed for the first time, and the to-be-updated video frame set is a video frame set output by the last execution of the dynamic updating strategy when the dynamic updating strategy is executed for the second time.

[0011] Optionally, the plurality of content evaluation indexes comprise: a first type of index for evaluating content coverage of a video frame and a second type of index for evaluating content difference of the video frame; wherein the first type of index is determined by: determining a target feature vector set corresponding to the candidate frame set, wherein the target feature vector set contains a plurality of target feature vectors, and each target feature vector is obtained by feature extraction on a candidate video frame; determining a similarity between each two target feature vectors in the target feature vector set, and constructing a target similarity matrix with the plurality of similarities as elements; and determining the first type of index according to a determinant value of the target similarity matrix.

[0012] Optionally, the second type of index is determined by: performing regularization processing on the target similarity matrix according to a preset regularization parameter and a unit matrix to obtain a regularized similarity matrix; determining an inverse matrix of the regularized similarity matrix, and determining a ridge leverage score of each candidate video frame according to the inverse matrix and the target similarity matrix, wherein the ridge leverage score is used to quantify the contribution of the candidate video frame to the diversity of the video to be sampled; and determining the second type of index according to the plurality of ridge leverage scores corresponding to all candidate video frames.

[0013] Optionally, determining the update result of the sampling parameter according to the plurality of content evaluation indexes comprises: obtaining a plurality of historical content evaluation indexes used when the dynamic update strategy is last executed, and determining a first difference between the content evaluation index and the historical content evaluation index evaluating the same information to obtain a plurality of first differences; determining a second difference between the information gain of the new video frame set and the information gain of the video frame set to be updated according to the historical content evaluation index and the content evaluation index; determining a function value of a reward function according to the second difference and the plurality of first differences, wherein the reward function is used to evaluate the change information of the first correlation compared with the second correlation, the first correlation is the relevance of the new video frame set to the video theme, and the second correlation is the relevance of the video frame set corresponding to the new video frame set to the video theme; and determining the update result of the sampling parameter according to the function value and the historical sampling parameter output when the dynamic update strategy is last executed.

[0014] According to a further aspect of the embodiments of the present application, a sampling device of video frames is also provided, comprising: a receiving module configured to receive a video to be sampled and a sampling requirement, wherein the sampling requirement records a preset sampling number, the preset sampling number being used to indicate a number of video frames extracted from the video to be sampled; a determining module configured to determine a first correlation index of each video frame contained in the video to be sampled, and to filter all the video frames contained in the video to be sampled according to the first correlation index of each video frame and the preset sampling number, to obtain an initial video frame set, wherein the first correlation index is used to quantitatively evaluate the correlation between the video frame and a video theme of the video to be sampled; an iterative updating module configured to iteratively perform a dynamic updating strategy on the video frame set to be updated until an iteration stopping condition is reached, and to output a target video frame set, wherein the target video frame set is the video frame set output by the last time of performing the dynamic updating strategy; wherein the dynamic updating strategy comprises: determining a first score of each candidate video frame contained in a candidate frame set, wherein the first score is used to quantitatively evaluate the correlation degree between the candidate video frame and the video theme, and the candidate frame set is a set composed of remaining video frames, the remaining video frames being the video frames in the video to be sampled and not contained in the video frame set to be updated; updating the video frame set to be updated according to the first score, the preset sampling number and the candidate frame set, to obtain a new video frame set; determining a plurality of content evaluation indexes of the new video frame set; determining an update result of a sampling parameter according to the plurality of content evaluation indexes, wherein the sampling parameter is used to guide the updating process of the video frame set to be updated; and outputting the new video frame set and the update result of the sampling parameter.

[0015] According to a further aspect of the embodiments of the present application, a non-volatile storage medium is also provided, the non-volatile storage medium storing a computer program, wherein a device in which the non-volatile storage medium is located executes the video frame sampling method described above by running the computer program.

[0016] According to a further aspect of the embodiments of the present application, an electronic device is also provided, comprising a memory and a processor, the memory storing a computer program, and the processor being configured to execute the video frame sampling method described above by the computer program.

[0017] According to a further aspect of the embodiments of the present application, a computer program product is also provided, comprising computer instructions, the computer instructions being executed by a processor to implement the steps of the video frame sampling method described above.

[0018] In the embodiment of the present application, the received video to be sampled and sampling requirements are adopted, wherein the preset sampling quantity is recorded in the sampling requirements, and the preset sampling quantity is used to indicate the quantity of video frames extracted from the video to be sampled; for each video frame contained in the video to be sampled, the first correlation index of each video frame is determined, and the first correlation index of each video frame and the preset sampling quantity are used to screen all video frames contained in the video to be sampled, so as to obtain an initial video frame set, wherein the first correlation index is used to quantitatively evaluate the correlation between the video frame and the video theme of the video to be sampled; the dynamic updating strategy is iteratively executed on the video frame set to be updated until the iteration stopping condition is reached, and the target video frame set is output. By introducing the ridge leverage score metric and volume metric, the content diversity and uniqueness of the video frame are quantified, and the Bayesian guided sequential sampling method is used to sample the key frame according to the correlation, diversity and uniqueness of the video frame, so as to achieve the purpose of considering the correlation and diversity of the video frame in the key frame sampling process, thereby realizing the technical effects of improving the correlation between the sampling result and the video theme and improving the accuracy of the sampling result. The target video frame set is the video frame set output by the last execution of the dynamic updating strategy, and the dynamic updating strategy includes: determining the first score of each candidate video frame contained in the candidate frame set, wherein the first score is used to quantitatively evaluate the correlation degree between the candidate video frame and the video theme, and the candidate frame set is a set composed of the remaining video frames, and the remaining video frames are the video frames in the video to be sampled which are not contained in the video frame set to be updated; updating the video frame set to be updated according to the first score, the preset sampling quantity and the candidate frame set to obtain a new video frame set; determining a plurality of content evaluation indexes of the new video frame set; determining the update result of the sampling parameter according to the plurality of content evaluation indexes, wherein the sampling parameter is used to guide the updating process of the video frame set to be updated; outputting the new video frame set and the update result of the sampling parameter; and dynamically adjusting the sampling parameter in the key frame sampling process according to the above dynamic updating strategy, so as to realize the purpose of improving the adaptability of the key frame sampling method to the application scene, thereby realizing the technical effects of enriching the application scene of the key frame sampling method and improving the efficiency of the key frame sampling, and further solving the technical problems of rigid sampling strategy, low sampling efficiency and low accuracy of the sampling result caused by the single factor of considering only the correlation of the video frame in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0020] Figure 1 is a hardware structure block diagram of a computer terminal for implementing a video frame sampling method according to an embodiment of the present application;

[0021] Figure 2 is a step flow chart of a sampling method of a video frame according to an embodiment of the present application;

[0022] Figure 3 is a structure diagram of a sampling device of a video frame according to an embodiment of the present application;

[0023] Figure 4 is a first correlation index of each video frame at a first time step according to an embodiment of the present application;

[0024] Figure 5 is a calculation result of information gain of each video frame at a first time step according to an embodiment of the present application;

[0025] Figure 6 is a score value of each video frame according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the personnel in the art better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0029] Similarity matrix: A two-dimensional matrix used to represent the similarity measure between keyframes. Each element in the matrix represents the similarity between two keyframes, which can be calculated in various ways such as Euclidean distance, Gaussian kernel function, etc. Generally, the larger the similarity value, the closer the two keyframes are in feature space, and the higher the similarity.

[0030] Volume measure: A method to measure the diversity of a set of keyframes by calculating the determinant of the similarity matrix. The larger the determinant, the weaker the linear correlation between the row vectors or column vectors of the similarity matrix, i.e., the more dispersed the distribution of keyframes in feature space, the greater the difference, thus reflecting higher diversity.

[0031] Ridge leverage score (RLS): Measures the uniqueness of each keyframe in feature space and its contribution to the diversity of the entire set of keyframes. By calculating the diagonal elements of the keyframe in the similarity matrix, it reflects the uniqueness of the keyframe in feature space. If the similarity of a keyframe with other keyframes is low, its corresponding ridge leverage score will be high, indicating that the keyframe contributes greatly to the diversity of the entire set.

[0032] Thompson sampling: A method used to balance between exploring new choices and exploiting known optimal choices. In the process of applying Thompson sampling, a posterior distribution is maintained for each choice, and a sample value is randomly selected according to the posterior distribution. As time goes on and feedback information increases, the posterior distribution is constantly updated, dynamically adjusting between exploration and exploitation.

[0033] Information gain (IG): Evaluates the diversity in the frame set by calculating the change in volume measure of the keyframe set after adding a certain keyframe. This change in volume measure is the information gain of the keyframe set, and the larger the change, the greater the improvement of the keyframe to the diversity of the set.

[0034] In related technologies, methods for keyframe sampling focus on improving the relevance of keyframes to ensure that the extracted frames accurately reflect the core content of the video. This method results in keyframe sampling results that are relatively single in content, lacking diversity and failing to comprehensively cover different scenes and details in the video. Some methods that pursue diversity can extract frames that cover different content, but may sacrifice the relevance of keyframes, making the extracted frames not closely related to the theme of the video; therefore, there are problems such as the difficulty of balancing relevance and diversity, rigid sampling strategies, etc., resulting in sampling results that do not fit the theme of the video and inaccurate sampling results. In order to solve this problem, the related solutions provided in the embodiments of the present application are described in detail below.

[0035] According to an embodiment of this application, a method embodiment for sampling video frames is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal for implementing a video frame sampling method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0038] The memory 104 can be used to store software programs of application software and modules, such as program instructions / data storage devices corresponding to the sampling method of video frames in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, i.e., implements the sampling method of video frames as described above. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include memories remotely arranged with respect to the processor 102, which can be connected to the computer terminal 10 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is used to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC) which can be connected to other network devices through a base station so as to be able to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0040] The display can be, for example, a touch screen type liquid crystal display (LCD) which can enable a user to interact with the user interface of the computer terminal 10.

[0041] The embodiments of the present application provide a sampling method of video frames which can be run in the above running environment, Figure 2 is a step flow chart of the sampling method of video frames provided by the embodiments of the present application, as shown in Figure 2 The method includes the following steps:

[0042] In step S202, a to-be-sampled video and a sampling requirement are received, wherein the sampling requirement records a preset sampling number, and the preset sampling number is used to indicate the number of video frames extracted from the to-be-sampled video.

[0043] The embodiment of the application provides a sequential sampling method based on Bayesian guidance. By introducing ridge leverage score and volume measurement, the diversity and uniqueness of video frames are quantified, and the correlation modeling is combined to realize the accurate balance of correlation and diversity. At the same time, the sampling strategy is dynamically adjusted by using Thompson Sampling in the sampling process, which improves the adaptability and sampling efficiency of video frame sampling to application scenarios. In step S202, a video to be sampled (i.e., a video to be sampled) and a sampling requirement of this time are received, wherein the sampling requirement is used to indicate a target of sampling the video to be sampled, and the target of sampling, such as the number of key frames extracted by sampling (i.e., a preset sampling number). In this embodiment, the video to be sampled carries a video topic embedding vector (which is in the form of a vector of video topics), which is an abstract representation of the content of the video to be sampled, reflecting the theme or core content of the video. It can be generated based on video title, description, label, etc.

[0044] In step S204, for each video frame included in the video to be sampled, a first correlation index of each video frame is determined, and each video frame is filtered in the entire video frame included in the video to be sampled according to the first correlation index of each video frame and the preset sampling number, to obtain an initial video frame set, wherein the first correlation index is used to quantify and evaluate the correlation of the video frame with the video topic of the video to be sampled.

[0045] The method provided by the embodiment of the application comprehensively considers the correlation and diversity of each video frame when performing key frame sampling, wherein the correlation refers to the degree of association of the video frame with the video topic of the video to be sampled, and in this embodiment, the correlation index is used to quantify the correlation of each video frame. In step S204, the correlation score (i.e., the first correlation index) of the video frame can be obtained by calculating the similarity between the video frame and the video topic embedding vector; next, based on the (first) correlation index of the video frame and the preset sampling number, an initial video frame set is selected from all video frames constituting the video to be sampled, wherein the number of video frames included in the initial video frame set is equal to the preset sampling number, and the initial key frame set is the basis for subsequent execution of the dynamic update strategy.

[0046] ​According to some optional embodiments of the present application, determining the first correlation index of each video frame comprises: determining a video theme of the video to be sampled, and encoding the video theme into a form of a vector to obtain a video theme embedding vector; and performing feature extraction on each video frame to obtain a feature vector of each video frame, wherein the feature vector records multi-dimensional information of the video frame, and the multi-dimensional information comprises visual information and text information of an image recorded by the video frame; for each video frame, determining a similarity between the feature vector and the video theme embedding vector, and determining the similarity as the first correlation index of the video frame.

[0047] The method provided by the embodiments of the present application evaluates the correlation of the video frame by utilizing the semantic association between the video frame and the video theme. Specifically, in the embodiments, the correlation score between the video frame and the video theme can be used as the (first) correlation index of the video frame, wherein the correlation score can be calculated by the following method: extracting a feature vector of each video frame in the video to be sampled, and performing interaction between the feature vector and a video theme embedding vector to obtain a correlation score between each video frame and the video theme, wherein the calculation formula of the correlation score (i.e., the correlation index, including the first correlation index and the second correlation index) is as follows: , wherein, represents a feature vector obtained by performing feature extraction on the video frame i by the CLIP model, is the transpose of , represents a video theme embedding vector of the video to be sampled, is the similarity (i.e., the correlation score) between and . The video theme embedding vector is obtained by encoding the video theme, and as mentioned in the above embodiments, the video theme is carried by the video to be sampled; the above-mentioned extraction of the feature vector of the video frame and the encoding of the video theme into the video theme embedding vector can be performed by a contrastive language-image pre-training (CLIP) model or other models with image-text processing functions.

[0048] According to some other optional embodiments of the present application, the initial video frame set is obtained by screening all the video frames contained in the to-be-sampled video according to the first correlation index of each video frame and the preset sampling number, including: determining a second score of each video frame according to the initial parameter and the first correlation index of each video frame, wherein each second score quantifies the degree to which each video frame is related to the video theme; sorting all the video frames according to the second scores of the video frames to obtain a video frame sequence, and screening the video frame sequence according to the arrangement order of the video frames in the video frame sequence and the preset sampling number to obtain the initial video frame set.

[0049] The initial key frame set is the object of the first execution of the dynamic update strategy, and is an initial video frame set that is highly related to the video theme and initially has diversity. In the present embodiment, the initial video frame set is generated mainly by relying on the input initial parameter. Specifically, first, the degree to which each video frame is related to the video theme is quantified by using the initial parameter and the correlation index of each video frame to obtain a correlation score (i.e., a second score) of each video frame. Next, the multiple video frames are arranged into a video frame sequence according to the second scores. In the present embodiment, when the video frame sequence is generated, the video frames can be arranged in order from high to low according to the second scores, or can be arranged in order from low to high according to the second scores. Finally, the preset sampling number of video frames are selected from the video frames according to the arrangement order of the video frames in the video frame sequence to form the initial video frame set. If the video frame sequence is generated by arranging the video frames in order from high to low according to the second scores, when the preset sampling number of video frames are selected from the video frames according to the arrangement order of the video frames in the video frame sequence, the top N video frames are selected starting from the highest ranking (i.e., the corresponding numerical value is the smallest, i.e., the arrangement order is the earliest), and N is the preset sampling number. If the video frame sequence is generated by arranging the video frames in order from low to high according to the second scores, when the preset sampling number of video frames are selected from the video frames according to the arrangement order of the video frames in the video frame sequence, the top N video frames are selected starting from the lowest ranking (i.e., the corresponding numerical value is the largest, i.e., the arrangement order is the earliest). The above-mentioned initial parameter is only used when the initial video frame set is generated, and in the non-first iteration process, a parameter of the same type as the initial parameter will be determined according to the actual information of the video frames.

[0050] Optionally, the initialization parameters comprise: preset initial dissimilarity and preset information gain ratio of each video frame; and the second score of each video frame is determined according to the initialization parameters and the first correlation index of each video frame, comprising: for each video frame, determining a first product of a first weight corresponding to the dissimilarity and the preset initial dissimilarity, a second product of a second weight corresponding to the correlation index and the first correlation index, and a third product of a third weight corresponding to the information gain ratio and the preset information gain ratio, and determining a sum of the first product, the second product and the third product as the second score.

[0051] The second score in the above embodiment is determined according to the formula: The calculation is performed, wherein, represents the score value (which can be the first score or the second score) of the video frame i, represents the dissimilarity of the video frame i, is the weight (i.e., the first weight) corresponding to the dissimilarity, which is used to indicate the importance of the dissimilarity to the score result, and the value thereof can be set according to actual requirements (such as 0.2); wherein, , wherein, is the feature vector of the video frame i, is the feature vector of the video frame j, is the historical video frame set (i.e., the video frame set contained in the output of the last execution of the dynamic updating strategy); is the (first) correlation index of the video frame i, is the weight (i.e., the second weight) corresponding to the correlation index, which is used to indicate the importance of the correlation index to the score result, and the value thereof can be set according to actual requirements (such as 0.5); is the information gain ratio, is the weight (i.e., the third weight) corresponding to the information gain ratio, which is used to indicate the importance of the information gain ratio to the score result, and the value thereof can be set according to actual requirements (such as 0.3). The above formula is applied in the generation process of the initial video frame set, and when calculating the (second) score of each video frame contained in the to-be-sampled video, the preset initial dissimilarity (for example, 1) is used to replace in the above formula; and the preset information gain ratio (such as 0.5) is used to replace in the above formula.

[0052] Step S206, iteratively performing the dynamic updating strategy on the to-be-updated video frame set until reaching an iteration stopping condition, and outputting a target video frame set, wherein the target video frame set is the video frame set output by the last execution of the dynamic updating strategy; wherein the dynamic updating strategy comprises: determining a first score of each candidate video frame included in a candidate frame set, wherein the first score is used to quantify the relevance of the candidate video frame to the video theme, and the candidate frame set is a set composed of remaining video frames, the remaining video frames being video frames in the to-be-sampled video that are not included in the to-be-updated video frame set; updating the to-be-updated video frame set according to the first score, the preset sampling number, and the candidate frame set to obtain a new video frame set; determining a plurality of content evaluation indexes of the new video frame set; determining an update result of a sampling parameter according to the plurality of content evaluation indexes, wherein the sampling parameter is used to guide the updating process of the to-be-updated video frame set; and outputting the new video frame set and the update result of the sampling parameter.

[0053] In step S206, the dynamic updating strategy is iteratively performed on the to-be-updated video frame set until reaching an iteration stopping condition, and a final result of key frame sampling performed on the to-be-sampled video (i.e., a target key frame set) is output, wherein the number of video frames included in the target key frame set is constrained by the preset sampling number obtained in step S202, and the iteration stopping condition can be that the number of iterations reaches a preset number, or that the change amount of the function value of the reward function is smaller than a preset change amount. In this embodiment, the object of the first (initial) execution of the dynamic updating strategy is the initial video frame set generated in step S204; in the iteration process, each execution of the dynamic updating strategy outputs a key frame sampling result of the to-be-sampled video, which will be used as the object of the next execution of the dynamic updating strategy, i.e., when the dynamic updating strategy is not executed for the first time, the to-be-updated video frame set is the video frame set included in the output result of the last iteration.

[0054] The dynamic updating strategy executed in step S206 includes the following steps: step 1, taking a set composed of all video frames contained in the to-be-sampled video as a universe, video frames belonging to the universe but not belonging to the to-be-updated video frame set are residual video frames, in the embodiment, a set composed of the residual video frames is referred to as a candidate frame set; for each video frame in the candidate frame set (i.e., a candidate video frame), a relevance score (i.e., a first score) of the candidate video frame with the video theme is calculated. Step 2, according to the first score of each candidate video frame, a preset sampling number of candidate video frames are selected from the candidate frame set, and a new video frame set composed of the preset sampling number of candidate video frames is a result of updating the to-be-updated video frame set in the current iteration process. Step 3, the sampling parameter is dynamically adjusted in the process of executing the dynamic updating strategy, wherein the sampling parameter is used to guide the updating process of the to-be-updated video frame set, specifically, after the sampling parameter is adjusted, the first score of each candidate video frame changes, and the generated new video frame set also changes naturally. In step 3, the sampling parameter is dynamically updated by using the diversity of the video frames, wherein in the embodiment, the diversity of the selected key frame set (such as the new video frame set) is quantified from the perspective of the feature space, specifically, a plurality of content evaluation indexes of the new video frame set are calculated, and the diversity of the new video frame set is quantified by using the content evaluation indexes. The plurality of content evaluation indexes refer to that the number of each content evaluation index is multiple, and the multiple content evaluation indexes respectively evaluate the video content contained in the new video frame set from different angles. Next, the updating result of the sampling parameter is determined by using the multiple content evaluation indexes. In the scheme provided in the embodiment of the application, the result output by each iteration includes a video frame set and a group of sampling parameters (the group of sampling parameters is the updating result of the sampling parameter applied when the current iteration executes the dynamic updating strategy).

[0055] According to some optional embodiments of the application, the first score of each candidate video frame contained in the candidate frame set is determined, including: obtaining the historical sampling parameter carried by the to-be-updated video frame set corresponding to the candidate video frame, wherein the historical sampling parameter is contained in the output of the dynamic updating strategy to which the to-be-updated video frame set belongs; determining a second relevance index of each candidate video frame, and a dissimilarity of the candidate video frame with the historical video frame set, wherein the historical video set is the video frame set output by the last execution of the dynamic updating strategy; and determining the first score of each candidate video frame according to the historical sampling parameter, the second relevance index and the dissimilarity.

[0056] As mentioned in the above embodiments, the same scoring formula is used to determine the first score and the second score: ; in the embodiment, when the above formula is applied to score the candidate video frames in the iteration process, is the feature vector of the candidate video frame i The dissimilarity of the historical video frame set is not replaced by the preset initial dissimilarity, but is calculated according to the formula is the relevance indicator (i.e., the second relevance indicator) of the candidate video frame i. When the above formula is applied in the iteration process for scoring the candidate video frame, is the information gain ratio of the candidate video frame i, can be calculated in combination with the formula and , where t represents the number of iterations (the number of rounds of iterations), represents the number of times that the video frame i is sampled, and the video frame i belongs to the number of times (maximum t) in t iteration results when the dynamic update strategy is executed t times. In this embodiment, the initial video set is generated once in one iteration; is the sampling parameter contained in the result of the (t-1)th iteration (i.e., the historical sampling parameter), represents a shape parameter distribution, is a probability distribution, the value of which is determined in a plurality of probability values contained in the probability distribution with the shape parameter . When calculating , t in is replaced by (t-1), and at this time, , and is the sampling parameter contained in the result of the (t-2)th iteration (i.e., the historical sampling parameter). In the scheme provided in this embodiment, the relevance indicator of the video frame contained in the video to be sampled is referred to as the first relevance indicator, and the relevance indicator of the candidate video frame is referred to as the second relevance indicator, and the first relevance indicator and the second relevance indicator are calculated using the same formula.

[0057] It should be noted that in the scheme provided in this embodiment, the first score and the second score can also be calculated using the following nonlinear formula; the nonlinear formula is: “use a ranking strategy based on dominance, define that video frame i dominates video frame j, and the condition is: at least one of the conditions is strictly true. At this time, by counting the number of times that each video frame is dominated by other video frames, the ranking priority of the video frame when generating a video frame sequence is determined; in the above formula, represents the information gain of the video frame i, ​​​information gain of a video frame j, the information gain is used to evaluate the diversity of a video frame set to which each video frame belongs. In the scheme provided by the embodiments of the present application, the first score and the second score can also be calculated by using the following vector formula: wherein, indicates the calculation of the determinant value, is a similarity matrix of the video frame set in the current round of iteration update result, the formula measures the volume gain of the key frame set after adding the video frame The greater the volume gain, the greater the contribution of the key frame to the diversity improvement.

[0058] Optionally, the to-be-updated video frame set is updated according to the first score, the preset sampling number and the candidate frame set to obtain a new video frame set, including: sorting all candidate video frames contained in the candidate frame set according to the first score corresponding to each candidate video frame to obtain a candidate video frame sequence; and screening the candidate frame set according to the sorting of the candidate video frames in the candidate video frame sequence and the preset sampling number to obtain the new video frame set, wherein the new video frame set is an update result of the to-be-updated video frame set, the to-be-updated video frame set is the initial video frame set when the dynamic update strategy is executed for the first time, and the to-be-updated video frame set is the video frame set output by the last execution of the dynamic update strategy when the dynamic update strategy is executed for the second time.

[0059] In this embodiment, in each iteration process, the update result (i.e., the new video frame set) of the to-be-updated video frame set is generated by the following steps. Specifically, similar to the process of generating the initial video frame set, the plurality of candidate video frames are arranged into a candidate video frame sequence according to the relevance scores (i.e., the first scores) of each candidate video frame. In this embodiment, when generating the candidate video frame sequence, the candidate video frames can be arranged in the order from high to low of the first scores, or can be arranged in the order from low to high of the first scores. Finally, the new video frame set is composed of the preset sampling number of candidate video frames selected from the candidate video frames according to the arrangement order of the candidate video frames in the candidate video frame sequence. If the candidate video frame sequence is generated by arranging the candidate video frames in the order from high to low of the first scores, when the preset sampling number of candidate video frames are selected from the candidate video frames according to the arrangement order of the candidate video frames in the candidate video frame sequence, the top N candidate video frames are selected from the top ranking (i.e., the value corresponding to the ranking is the smallest, i.e., the arrangement order is the earliest), and N is the preset sampling number. If the candidate video frame sequence is generated by arranging the candidate video frames in the order from low to high of the first scores, when the preset sampling number of candidate video frames are selected from the candidate video frames according to the arrangement order of the candidate video frames in the candidate video frame sequence, the top N candidate video frames are selected from the lowest ranking (i.e., the value corresponding to the ranking is the largest, i.e., the arrangement order is the earliest). In the scheme provided in this embodiment of the application, only when the dynamic update strategy is executed for the first time, the object of executing the dynamic update strategy is the initial video frame set. Except for the first time of executing the dynamic update strategy, the to-be-updated video frame set is the video frame set output by the last execution of the dynamic update strategy.

[0060] According to some optional embodiments of the application, the plurality of content evaluation indexes include: a first type of index for evaluating the content coverage of the video frame, and a second type of index for evaluating the content difference of the video frame; wherein the first type of index is determined by the following method: determining a target feature vector set corresponding to the candidate frame set, wherein the target feature vector set includes a plurality of target feature vectors, and each target feature vector is obtained by performing feature extraction on a candidate video frame; determining the similarity of each two target feature vectors in the target feature vector set, and constructing a target similarity matrix with the plurality of similarities as elements; and determining the first type of index according to the determinant value of the target similarity matrix.

[0061] The content evaluation index used in the embodiments of the present application includes: a volume metric score (Volume) for evaluating the content coverage of the video frame set (i.e., the first type of index), and a ridge leverage score (RLS) for evaluating the content difference of the video frame (i.e., the second type of index). The volume metric score is obtained by constructing a similarity matrix and calculating the determinant value of the similarity matrix. In the embodiments of the present application, the content diversity of the candidate frame set is measured by calculating the volume metric score of the candidate frame set. The volume metric score (i.e., the first type of index) can be calculated by the following formula: wherein S represents the candidate frame set, is the similarity matrix (i.e., the target similarity matrix) of the candidate frame set , e represents the logarithmic operation with e (e≈2.71828), indicates the determinant value. The elements of the similarity matrix are defined as: wherein and are the feature vectors (i.e., the target feature vectors) of the video frame and the video frame , respectively, and the video frame and the video frame are two video frames in the candidate frame set, is the bandwidth parameter of the Gaussian kernel, used to control the range of the similarity metric, is the similarity of the feature vectors (i.e., the target feature vectors) of the video frame and the video frame .

[0062] In the embodiments, the volume metric can capture the distribution and coverage range of the candidate frame set in the feature space, and ensure that the selected key frames can be as dispersed as possible in the feature space, thereby improving the diversity of the key frame set. The calculation of the determinant reflects the “volume” of the matrix, i.e., the spatial size occupied by the video frames in the feature space. The larger the determinant value is, the greater the content difference between the video frames is, and the higher the diversity is. The larger the determinant value is, the weaker the linear correlation between the row vectors or the column vectors of the similarity matrix is, i.e., the more dispersed the distribution of the video frames in the feature space is, and the greater the difference is. At this time, the video frame set can cover more regions of the feature space, thereby reflecting higher diversity. For example, if multiple video frames are very close in the feature space, the similarity between them is high, and the determinant of the corresponding similarity matrix is small. On the contrary, if the video frames are dispersed in different regions of the feature space, the similarity between them is low, and the determinant of the corresponding similarity matrix is large. In this way, the volume metric can effectively reflect the diversity of the video frame set. ​

[0063] According to some alternative embodiments of this application, the second type of index is determined by the following method: the target similarity matrix is ​​regularized according to a preset regularization parameter and an identity matrix to obtain a regularized similarity matrix; the inverse matrix of the regularized similarity matrix is ​​determined, and the ridge lever score of each candidate video frame is determined according to the inverse matrix and the target similarity matrix, wherein the ridge lever score is used to quantify the contribution of the candidate video frame to the diversity of the video to be sampled; the second type of index is determined according to the multiple ridge lever scores corresponding to all candidate video frames.

[0064] Ridge Leverage Score: The ridge leverage score (or ridge leverage ratio) measures the uniqueness (i.e., content diversity) of each video frame in the feature space and its contribution to the diversity of the set of video frames it belongs to. The ridge leverage score measures cross-batch diversity by calculating each video frame's contribution to the context. The formula for calculating the ridge leverage score is:

[0065]

[0066]

[0067] in, It is the set of video frames to be updated, and I is the video frame selected as output in the current iteration. yes and union, Indicates the set of video frames to be updated and new video frames The similarity matrix, The preset regularization parameter is used to prevent numerical instability during matrix inversion. ) is the regularized similarity matrix, It is the inverse of the regularized similarity matrix. It is the first The ridge leverage score is calculated for each video frame. The core idea of ​​the ridge leverage score metric is to reflect the uniqueness of a keyframe in the feature space by calculating the diagonal elements of the video frame in the similarity matrix. If a keyframe has low similarity to other keyframes, its corresponding ridge leverage score will be high, indicating that the keyframe contributes significantly to the diversity of the entire set. It is the ridge lever score of the candidate frame set S.

[0068] Optionally, the update result of the sampling parameter is determined according to a plurality of content evaluation indexes, including: obtaining a plurality of historical content evaluation indexes used when the dynamic update strategy is executed last time, and determining a first difference value of the content evaluation index and the historical content evaluation index evaluating the same information, to obtain a plurality of first difference values; determining a second difference value of information gain of the new video frame set and information gain of the video frame set to be updated according to the historical content evaluation index and the content evaluation index; determining a function value of a reward function according to the second difference value and the plurality of first difference values, wherein the reward function is used to evaluate the change information of the first correlation compared with the second correlation, the first correlation is the correlation of the new video frame set and the video theme, and the second correlation is the correlation of the video frame set corresponding to the new video frame set and the video theme; and determining the update result of the sampling parameter according to the function value and the historical sampling parameter output when the dynamic update strategy is executed last time.

[0069] The method provided by the embodiment of the application sets a reward function reward, which comprehensively considers the volume metric score, the ridge leverage metric score and the change of information gain, ensures the consistency of diversity gain, and prevents mutual offset. The calculation formula of the function value of the reward function reward is:

[0070]

[0071] wherein, , , and belong to the first difference value; represents the change amount of the average value of the information gain at the current time step (i.e. the current iteration round), is the difference value between the average value of the information gain of the video frame set to be updated and the average value of the information gain of the new video frame set; , , is a weight coefficient, used to balance the influence of different factors on the function value of the reward function. The information gain (IG) is calculated by using the formula , wherein, is the current candidate video frame set, and the candidate frame with larger information gain has higher exploration value. Next, the sampling parameter is updated according to Thompson Sampling and , and is updated based on the positive and negative values of the reward function, and the update formula is as follows: .

[0072] Through the above steps, the diversity of key frames can be accurately measured in the sampling process, ensuring that the extracted video frames comprehensively cover different contents of the video to be sampled, and avoiding the selection of redundant frames; the volume metric score, the ridge leverage score and the change of information gain are comprehensively considered to accurately measure the gain in the key frame sampling process, providing a reliable basis for dynamically adjusting the sampling parameters; the Thompson Sampling is used to dynamically adjust the sampling strategy, which can flexibly balance between exploring new key frames and using known key frames according to the current extraction situation and known information, improving the adaptability of the video frame sampling method to application scenarios, and enabling it to better cope with different types of videos and complex scene changes.

[0073] Figure 3 is a structural diagram of a video frame sampling device provided by an embodiment of the present application, as shown in Figure 3 the video frame sampling device includes: a receiving module 30, configured to receive a video to be sampled and sampling requirements, wherein the sampling requirements record a preset sampling number, which is used to indicate the number of video frames extracted from the video to be sampled; a determining module 32, configured to determine a first correlation index of each video frame contained in the video to be sampled, and perform screening among all video frames contained in the video to be sampled according to the first correlation index of each video frame and the preset sampling number, to obtain an initial video frame set, wherein the first correlation index is used to quantitatively evaluate the correlation between the video frame and the video theme of the video to be sampled; an iterative updating module 34, configured to iteratively execute a dynamic updating strategy on the video frame set to be updated until an iteration stopping condition is reached, and output a target video frame set, wherein the target video frame set is the video frame set output by the last execution of the dynamic updating strategy; wherein the dynamic updating strategy includes: determining a first score of each candidate video frame contained in a candidate frame set, wherein the first score is used to quantitatively evaluate the correlation degree between the candidate video frame and the video theme, and the candidate frame set is a set composed of remaining video frames, the remaining video frames being the video frames in the video to be sampled that are not contained in the video frame set to be updated; updating the video frame set to be updated according to the first score, the preset sampling number and the candidate frame set, to obtain a new video frame set; determining a plurality of content evaluation indexes of the new video frame set; determining an update result of a sampling parameter according to the plurality of content evaluation indexes, wherein the sampling parameter is used to guide the updating process of the video frame set to be updated; and outputting the new video frame set and the update result of the sampling parameter.

[0074] The video frame sampling device executes the execution process of the video frame sampling method provided by the present application as follows: the input information received by the receiving module 30 is: a video sequence , a key frame number (i.e. a preset sampling number), a video theme embedding vector , and a video frame feature matrix (Where is the number of frames contained in the video sequence, is the feature dimension). Stopping condition: monitor the reward function value or other content indicators (such as volume metric, ridge leverage score metric, etc.) after each sampling, if it is found that the sampling of new key frames cannot significantly improve the relevance or diversity of the key frame set, that is, the change of the reward function value tends to be stable or no longer increases, the sampling process can be stopped. Output: key frame set (i.e. target video frame set) The determination module 32 performs an initialization process: the selected key frame set (empty set), the count of each key frame , the ThompsonSampling parameter , , the initial gain , the volume metric score of the last round (OldVolume), the ridge leverage score of the last round (OldRLS), and the information gain of the last round (OldIG) are initialized to zero. The iterative update module 34 performs sequential sampling: at each time step (that is, each time sampling is performed, each iteration), the batch selection process is called, and the ridge leverage score (RLS), volume metric score (Volume), and information gain (IG) score of each key frame are calculated using the following formula:

[0075]

[0076]

[0077]

[0078] Further, the score of each video frame is calculated according to the current parameters, and the candidate frame set with the highest score is selected. The reward is calculated and the and parameters are updated. Repeat the process until the maximum number of samplings is reached or other termination conditions are met. After each sampling, the score and ranking of the key frames are recalculated according to the updated parameters, and the key frame set is gradually optimized.

[0079] It should be noted that Figure 3 the preferred embodiments of the embodiments shown in Figure 2 can be referred to the related description of the embodiments shown in, which will not be repeated here.

[0080] Next, the sampling method of the video frame provided by the embodiments of the present application is illustrated by example. The video to be sampled contains 10 video frames, and the video theme embedding vector The feature vectors for each video frame are as follows: Video frame 1 feature vector [1,2], Video frame 2 feature vector [2,3], Video frame 3 feature vector [3,4], Video frame 4 feature vector [4,5], Video frame 5 feature vector [6,7], Video frame 6 feature vector [7,8], Video frame 7 feature vector [8,9], Video frame 8 feature vector [9,10], Video frame 9 feature vector [10,11], Video frame 10 feature vector [11,12]. Initialization parameters include: the number of keyframes (i.e., the preset sampling number). Video topic embedding vector The selected set of keyframes (each output video frame is a keyframe) Count of each keyframe ThompsonSampling parameters , initial gain OldVolume and OldRLS are initialized to zero. Figure 4 It is the first relevance metric for each video frame at the first time step. That is, when the dynamic update strategy is executed for the first time, firstly, according to the formula... Calculate the relevance score for each video frame. ,get Figure 4 The results are shown. Figure 5 It is the information gain of each video frame at the first time step. Next, according to the formula Calculate the information gain for each video frame i. ,get Figure 5 The results show that the information gain of the video frames participates in the updating of the sampling parameters. Since the initial set is empty at the first time step, the information gain of each keyframe as a candidate subset is calculated. The information gain of each video frame is calculated by constructing a similarity matrix. For example, when calculating the information gain of video frame 1, a similarity matrix is ​​constructed. Assuming the determinant is 50, then: Information gain: When calculating the information gain of video frame 1, a similarity matrix is ​​constructed. Assuming the determinant is 70, then: Information gain: And so on, calculate the information gain of all video frames to obtain... Figure 5 The results are shown. Figure 6 It is the score value for each video frame, calculated after the first relevance index, according to the linear scoring formula. Calculate the score of the video frame and select the weighting coefficient. , , , initial time instant dissimilarity score = 1, gain ratio = 0.5 (initial setting for simplifying the first calculation), resulting in the outcome shown in FIG. 4. After calculating the score of each video frame, the scores are sorted from high to low, and the top 3 keyframes with the highest scores are selected as the initial candidate subset, according to the score number shown in FIG. 5, the initial candidate subset is Figure 6 . Further, multiple content indicators (volume metric score and ridge leverage score) of the candidate video set are calculated, wherein, when calculating the volume metric score, the similarity matrix Figure 6 of the candidate subset is constructed, assuming that the determinant is 300, then: . Next, the ridge leverage score of each video frame is calculated, assuming that , , , , , then the ridge leverage metric score ( ) . Next, the function value of the reward function (weight coefficient ) is calculated according to the reward function formula: , , . Since the initial set is empty, the initial information gain is zero, so: (avg represents the average value), the average information gain of the candidate subset is: , and the function value of the reward function is: . Finally, the sampling parameter is updated: the Thompson Sampling parameter is updated according to the reward value. Since is positive, for each keyframe in , increases , that is, , and remains 1.

[0081] When sampling video frames at the second time step , the input parameters include: the video frame set to be updated (i.e., the video frame set output by the last iteration) , the updated Thompson Sampling parameter , , . At the second time step, first, the score of each keyframe is recalculated, considering the new keyframe set : dissimilarity score : Compute dissimilarity of each keyframe from the set , assuming the result as follows: Frame 1: 0.8, Frame 2: 0.7, Frame 3: 0.6, Frame 4: 0.5, Frame 5: 0.4, Frame 6: 0.3, Frame 7: 0.2; gain ratio : Frame 1: 0.4, Frame 2: 0.3, Frame 3: 0.2, Frame 4: 0.1, Frame 5: 0.05, Frame 6: 0.03, Frame 7: 0.02. Recompute scores according to the linear score formula, e.g. for Frame 7: . Sort according to the new scores, select the top 3 keyframes as the new candidate subset, e.g. . Compute new volume metric and ridge leverage score, where the volume metric: construct the similarity matrix of the new candidate subset , assuming the determinant is 350, then: ; compute the new ridge leverage score, assuming they are: , , , the new RLS score is: . Compute the information gain of each keyframe in the new set: , assuming , then: , similarly, compute and , assuming they are -0.301 and -0.223 respectively. The new average information gain: . Compute the new reward . Finally, update the parameters, since is negative, for each keyframe in , is increased by , i.e. , while remains unchanged. At this time, the stopping condition is reached, and no more iterations are performed. It should be noted that in the logarithm (log) operation appearing in the above, e (e 2.71828) is taken as the base number, when the logarithm operation is taken with e as the base number, it can be written in the form of log , and also can be written in the form of .

[0082] The embodiment of the application further provides a non-volatile storage medium, and the non-volatile storage medium stores a computer program, wherein a device in which the non-volatile storage medium is located executes the above-mentioned video frame sampling method by running the computer program.

[0083] The nonvolatile storage medium is configured to store a program for receiving a video to be sampled and a sampling requirement, wherein the sampling requirement records a preset sampling number, and the preset sampling number is used to indicate a number of video frames extracted from the video to be sampled; determining a first correlation index of each video frame included in the video to be sampled, and screening all video frames included in the video to be sampled according to the first correlation index of each video frame and the preset sampling number to obtain an initial video frame set, wherein the first correlation index is used to quantitatively evaluate the correlation between the video frame and a video theme of the video to be sampled; iteratively performing a dynamic update strategy on the updated video frame set until an iteration stop condition is reached, and outputting a target video frame set, wherein the target video frame set is a video frame set output by the last execution of the dynamic update strategy; wherein the dynamic update strategy comprises: determining a first score of each candidate video frame included in a candidate frame set, wherein the first score is used to quantitatively evaluate the correlation between the candidate video frame and the video theme, and the candidate frame set is a set composed of remaining video frames, and the remaining video frames are video frames in the video to be sampled that are not included in the updated video frame set; updating the updated video frame set according to the first score, the preset sampling number and the candidate frame set to obtain a new video frame set; determining a plurality of content evaluation indexes of the new video frame set; determining an update result of a sampling parameter according to the plurality of content evaluation indexes, wherein the sampling parameter is used to guide the update process of the updated video frame set; and outputting the new video frame set and the update result of the sampling parameter.

[0084] The embodiments of the present application also provide an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to execute the above video frame sampling method through the computer program.

[0085] The processor in the electronic device is configured to run a program for performing the following functions: receiving a video to be sampled and a sampling requirement, wherein the sampling requirement records a preset sampling number, and the preset sampling number is used to indicate a number of video frames extracted from the video to be sampled; determining a first correlation index of each video frame included in the video to be sampled, and performing screening on all video frames included in the video to be sampled according to the first correlation index of each video frame and the preset sampling number, to obtain an initial video frame set, wherein the first correlation index is used to quantitatively evaluate the correlation between the video frame and a video theme of the video to be sampled; iteratively performing a dynamic updating strategy on the updated video frame set until an iteration stopping condition is reached, and outputting a target video frame set, wherein the target video frame set is a video frame set output by the last execution of the dynamic updating strategy; wherein the dynamic updating strategy comprises: determining a first score of each candidate video frame included in a candidate frame set, wherein the first score is used to quantitatively evaluate the correlation degree between the candidate video frame and the video theme, and the candidate frame set is a set composed of remaining video frames, and the remaining video frames are video frames in the video to be sampled that are not included in the updated video frame set; updating the updated video frame set according to the first score, the preset sampling number and the candidate frame set, to obtain a new video frame set; determining a plurality of content evaluation indexes of the new video frame set; determining an update result of a sampling parameter according to the plurality of content evaluation indexes, wherein the sampling parameter is used to guide the updating process of the updated video frame set; and outputting the new video frame set and the update result of the sampling parameter.

[0086] The embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the above video frame sampling method.

[0087] It should be noted that each module in the above video frame sampling device can be a program module (for example, a program instruction set for implementing a certain specific function) or a hardware module. For the latter, it can be in the following form, but is not limited to this: the form of each module is a processor, or the functions of each module are implemented by a processor.

[0088] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0089] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0090] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division mode, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0091] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0092] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0093] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the related art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0094] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A method for sampling video frames, characterized in that, include: Receive a video to be sampled and a sampling request, wherein the sampling request contains a preset sampling number, which is used to indicate the number of video frames extracted from the video to be sampled; For each video frame contained in the video to be sampled, a first correlation index is determined for each video frame, and the first correlation index of each video frame and the preset sampling number are used to filter all video frames contained in the video to be sampled to obtain an initial set of video frames, wherein the first correlation index is used to quantitatively evaluate the correlation between the video frame and the video theme of the video to be sampled; A dynamic update strategy is iteratively executed on the set of video frames to be updated until the iteration stops, and a target set of video frames is output. When the dynamic update strategy is executed for the first time, the set of video frames to be updated is the initial set of video frames. When the dynamic update strategy is executed again, the set of video frames to be updated is the set of video frames output from the previous execution of the dynamic update strategy. The target set of video frames is the set of video frames output from the last execution of the dynamic update strategy. The dynamic update strategy includes: determining a first score for each candidate video frame in the candidate frame set, wherein the first score is used to quantify the relevance of the candidate video frame to the video topic, the candidate frame set is a set of remaining video frames, the remaining video frames being video frames in the video to be sampled that are not included in the video frame set to be updated; updating the video frame set to be updated according to the first score, the preset sampling quantity, and the candidate frame set to obtain a new video frame set; determining multiple content evaluation metrics for the new video frame set; determining the update result of the sampling parameters according to the multiple content evaluation metrics, wherein the sampling parameters are used to guide the update process of the video frame set to be updated; and outputting the update result of the new video frame set and the sampling parameters; the internal The content evaluation metrics include: a second type of metric for evaluating the content differences of video frames, wherein the second type of metric is determined by the following method: regularizing the target similarity matrix according to a preset regularization parameter and an identity matrix to obtain a regularized similarity matrix, wherein the target similarity matrix is ​​constructed using the similarity of the two target feature vectors corresponding to every two candidate video frames in the candidate frame set as elements; determining the inverse matrix of the regularized similarity matrix, and determining the ridge lever score of each candidate video frame according to the inverse matrix and the target similarity matrix, wherein the ridge lever score is used to quantify the contribution of the candidate video frame to the diversity of the video to be sampled; and determining the second type of metric based on multiple ridge lever scores corresponding to all the candidate video frames.

2. The method according to claim 1, characterized in that, Determining a first relevance metric for each of the video frames includes: Determine the video theme of the video to be sampled, and encode the video theme into a vector form to obtain a video theme embedding vector; and, Feature extraction is performed on each video frame to obtain a feature vector for each video frame. The feature vector contains multidimensional information of the video frame, including visual information and text information of the image recorded in the video frame. For each video frame, the similarity between the feature vector and the video topic embedding vector is determined, and the similarity is determined as the first relevance index of the video frame.

3. The method according to claim 1, characterized in that, Based on the first relevance index of each video frame and the preset sampling number, an initial set of video frames is obtained by filtering all video frames contained in the video to be sampled, including: A second score is determined for each video frame based on initialization parameters and a first relevance index for each video frame, wherein each second score is used to quantify the degree to which each video frame is relevant to the video topic; All video frames are sorted according to the second score of the video frames to obtain a video frame sequence. The video frame sequence is then filtered according to the order of the video frames in the video frame sequence and the preset sampling number to obtain the initial video frame set.

4. The method according to claim 3, characterized in that, The initialization parameters include: a preset initial dissimilarity and a preset information gain ratio for each video frame; a second score for each video frame is determined based on the initialization parameters and a first relevance index for each video frame, including: For each video frame, the product of the first weight corresponding to the dissimilarity and the first product of the preset initial dissimilarity, the product of the second weight corresponding to the relevance index and the second product of the first relevance index, and the product of the third weight corresponding to the information gain ratio and the third product of the preset information gain ratio are determined. The sum of the first product, the second product and the third product is determined as the second score.

5. The method according to claim 1, characterized in that, Determine the first score for each candidate video frame in the candidate frame set, including: Obtain the historical sampling parameters carried by the set of video frames to be updated corresponding to the candidate video frames, wherein the historical sampling parameters are included in the output of the dynamic update strategy to which the set of video frames to be updated belongs; A second relevance index is determined for each candidate video frame, and the dissimilarity between the candidate video frame and the historical video frame set is determined, wherein the historical video frame set is the set of video frames output by the last execution of the dynamic update strategy; a first score is determined for each candidate video frame based on the historical sampling parameters, the second relevance index, and the dissimilarity.

6. The method according to claim 1, characterized in that, The set of video frames to be updated is updated based on the first score, the preset number of samples, and the candidate frame set to obtain a new set of video frames, including: Sort all the candidate video frames in the candidate frame set according to the first score corresponding to each candidate video frame to obtain a candidate video frame sequence; The candidate frame set is filtered according to the order of the candidate video frames in the candidate video frame sequence and the preset sampling number to obtain the new video frame set, wherein the new video frame set is the update result of the video frame set to be updated.

7. The method according to claim 1, characterized in that, The content evaluation metrics further include: a first type of metric for evaluating the content coverage of video frames; wherein the first type of metric is determined by the following method: Determine the target feature vector set corresponding to the candidate frame set, wherein the target feature vector set contains multiple target feature vectors, and each target feature vector is obtained by feature extraction from a candidate video frame; Determine the similarity between every two target feature vectors in the target feature vector set, and construct a target similarity matrix using multiple similarity values ​​as elements; Determine the determinant value of the target similarity matrix, and determine the first type of index based on the determinant value.

8. The method according to claim 1, characterized in that, The updated sampling parameters are determined based on several of the aforementioned content evaluation metrics, including: The system obtains multiple historical content evaluation metrics used when the dynamic update strategy was last executed, and determines a first difference between the content evaluation metrics used to evaluate the same information and the historical content evaluation metrics, thereby obtaining multiple first differences. A second difference between the information gain of the new video frame set and the information gain of the video frame set to be updated is determined based on the historical content evaluation index and the content evaluation index. The reward function value is determined based on the second difference and multiple first differences, wherein the reward function is used to evaluate the change information of the first correlation relative to the second correlation, the first correlation being the correlation between the new video frame set and the video theme, and the second correlation being the correlation between the video frame set to be updated corresponding to the new video frame set and the video theme; The update result of the sampling parameters is determined based on the function value and the historical sampling parameters output from the last execution of the dynamic update strategy.

9. A video frame sampling device, characterized in that, include: A receiving module is used to receive a video to be sampled and a sampling requirement, wherein the sampling requirement contains a preset sampling quantity, which is used to indicate the number of video frames extracted from the video to be sampled; The determining module is used to determine a first correlation index for each video frame contained in the video to be sampled, and to filter all video frames contained in the video to be sampled according to the first correlation index of each video frame and the preset sampling number to obtain an initial set of video frames, wherein the first correlation index is used to quantitatively evaluate the correlation between the video frame and the video theme of the video to be sampled; The iterative update module is used to iteratively execute a dynamic update strategy on the set of video frames to be updated until the iteration stops, and output a target set of video frames. Specifically, when the dynamic update strategy is executed for the first time, the set of video frames to be updated is the initial set of video frames; when the dynamic update strategy is executed again, the set of video frames to be updated is the set of video frames output from the previous execution of the dynamic update strategy; and the target set of video frames is the set of video frames output from the last execution of the dynamic update strategy. The dynamic update strategy includes: determining a first score for each candidate video frame in the candidate frame set, wherein the first score is used to quantify the relevance of the candidate video frame to the video topic, the candidate frame set is a set of remaining video frames, the remaining video frames being video frames in the video to be sampled that are not included in the video frame set to be updated; updating the video frame set to be updated according to the first score, the preset sampling quantity, and the candidate frame set to obtain a new video frame set; determining multiple content evaluation metrics for the new video frame set; determining the update result of the sampling parameters according to the multiple content evaluation metrics, wherein the sampling parameters are used to guide the update process of the video frame set to be updated; and outputting the update result of the new video frame set and the sampling parameters; the internal The content evaluation metrics include: a second type of metric for evaluating the content differences of video frames, wherein the second type of metric is determined by the following method: regularizing the target similarity matrix according to a preset regularization parameter and an identity matrix to obtain a regularized similarity matrix, wherein the target similarity matrix is ​​constructed using the similarity of the two target feature vectors corresponding to every two candidate video frames in the candidate frame set as elements; determining the inverse matrix of the regularized similarity matrix, and determining the ridge lever score of each candidate video frame according to the inverse matrix and the target similarity matrix, wherein the ridge lever score is used to quantify the contribution of the candidate video frame to the diversity of the video to be sampled; and determining the second type of metric based on multiple ridge lever scores corresponding to all the candidate video frames.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, wherein the device containing the non-volatile storage medium executes the video frame sampling method according to any one of claims 1 to 8 by running the computer program.

11. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the video frame sampling method according to any one of claims 1 to 8 through the computer program.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the video frame sampling method according to any one of claims 1 to 8.

Images