Video thumbnail extraction method, device and equipment

Abstract

The invention provides a video thumbnail extraction method, device and equipment, and relates to the technical field of image processing, and the method comprises the steps: carrying out the attention fusion of initial video features of a to-be-processed video, and obtaining the fusion video features corresponding to a plurality of video frames of the to-be-processed video, the initial video features are obtained by performing feature extraction on a to-be-processed video, and the initial video features comprise initial image features, initial audio features and initial text features; processing the fused video features based on a Cauchy distribution mechanism to obtain distribution parameters corresponding to the plurality of video frames, the distribution parameters representing differences between frame contents of the video frames and frame contents of a plurality of other video frames in the to-be-processed video; performing scoring calculation on the respective distribution parameters of the plurality of video frames to obtain a scoring calculation result; analyzing the score calculation result based on a time uniform distribution condition to obtain a plurality of target video frames; and fusing the plurality of target video frames to obtain a video thumbnail of the to-be-processed video.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method, apparatus, and device for extracting video thumbnails. Background Technology

[0002] Most video thumbnail extraction methods extract image features such as color distribution, texture features, and edge information of each video frame in order to determine the importance of each video frame based on these image features.

[0003] However, the above methods are not suitable for videos with complex backgrounds, are easily affected by noise and background changes, resulting in low accuracy of extracted image features and difficulty in flexibly adapting to different types of videos. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method, apparatus and device for extracting video thumbnails.

[0005] According to a first aspect of the present invention, a video thumbnail extraction method is provided, comprising: performing attention fusion on initial video features of a video to be processed to obtain fused video features corresponding to multiple video frames of the video to be processed, wherein the initial video features are obtained by feature extraction of the video to be processed, and the initial video features include initial image features, initial audio features, and initial text features; processing the fused video features based on the Cauchy distribution mechanism to obtain distribution parameters corresponding to multiple video frames, wherein the distribution parameters represent the differences between the frame content of the video frame and the frame content of multiple other video frames in the video to be processed; performing scoring calculation on the distribution parameters of the multiple video frames to obtain a scoring calculation result; analyzing the scoring calculation result based on a time uniform distribution condition to obtain multiple target video frames; and fusing the multiple target video frames to obtain a video thumbnail of the video to be processed.

[0006] According to an embodiment of the present invention, the distribution parameters include position parameters and scale parameters. The position parameters represent the importance of the frame content of the video frame, and the scale parameters represent the information density of the frame content of the video frame. A scoring calculation is performed on the distribution parameters of multiple video frames to obtain a scoring result, including: for any one of the multiple video frames, determining a score value based on the reciprocal of the scale parameter; selecting the distribution parameters of the multiple video frames that satisfy a preset distribution parameter threshold as target distribution parameters, and selecting the video frame corresponding to the target distribution parameter as a first candidate video frame; and determining the scoring result based on the score value corresponding to the first candidate video frame.

[0007] According to an embodiment of the present invention, the video frame includes time period information; the scoring calculation results are analyzed based on the time uniform distribution condition to obtain multiple target video frames, including: obtaining a second candidate video frame based on the score value corresponding to the first candidate video frame according to the preset screening conditions; and selecting at least one second candidate video frame as a target video frame in each preset time period according to the time attributes of the multiple second candidate video frames.

[0008] According to an embodiment of the present invention, attention fusion is performed on video features to obtain fused video features for each of the multiple video frames of the video to be processed, including: for any one of the multiple video frames, weighted fusion is performed on the initial image features, initial audio features and initial text features at different scales according to their respective attention weights to obtain the fused video features of the video frame.

[0009] According to an embodiment of the present invention, the above method further includes: performing attention fusion on sample video features based on attention weights to obtain sample fused video features for each of multiple sample video frames of the sample video, wherein the sample video features are obtained by feature extraction of the sample video; processing the sample fused video features based on the Cauchy distribution mechanism to obtain sample distribution parameters for each of the multiple sample video frames; calculating a score for each of the sample distribution parameters of the multiple sample video frames to obtain a sample score calculation result; analyzing the sample score calculation result based on the time uniform distribution condition to obtain multiple target sample video frames; optimizing the attention weights according to the loss value between the target sample video frames and the label video frames to obtain optimized attention weights; wherein, weighted fusion of initial image features, initial audio features, and initial text features at different scales is performed based on the attention weights of the initial image features, initial audio features, and initial text features, including: weighted fusion of initial image features, initial audio features, and initial text features at different scales based on the optimized attention weights of the initial image features, initial audio features, and initial text features.

[0010] According to an embodiment of the present invention, the fused video features are processed based on the Cauchy distribution mechanism to obtain the distribution parameters corresponding to each of the multiple video frames, including: updating the probability density function of the Cauchy distribution based on the fused video features to obtain the likelihood function; performing a natural logarithm calculation operation on the likelihood function to obtain the target likelihood function; and solving the target likelihood function based on a preset solution objective to obtain the distribution parameters, wherein the preset solution objective indicates that the solution value of the target likelihood function satisfies a preset solution condition.

[0011] According to an embodiment of the present invention, fusing multiple target video frames to obtain a video thumbnail of the video to be processed includes: stitching multiple target video frames together to obtain an initial video thumbnail; and performing image enhancement on the initial video thumbnail to obtain a video thumbnail of the video to be processed.

[0012] According to an embodiment of the present invention, feature extraction of the video to be processed includes: sampling the video to be processed to obtain a specified number of video frames per second; extracting features from the color values ​​of pixels in the video frames, texture information sequences associated with the video frames, and position information of the target object based on a local binary extraction algorithm to obtain initial image features, wherein the texture information sequence represents the arrangement result of pixel texture information of multiple pixels; extracting features from the audio signals of the video frames based on a frequency domain extraction algorithm to obtain initial audio features; and extracting features from the text information of the video frames based on a text recognition algorithm to obtain initial text features.

[0013] A second aspect of the present invention provides a video thumbnail extraction apparatus, comprising: a first fusion module, configured to perform attention fusion on initial video features of a video to be processed to obtain fused video features corresponding to multiple video frames of the video to be processed, wherein the initial video features are obtained by feature extraction of the video to be processed, and the initial video features include initial image features, initial audio features, and initial text features; a processing module, configured to process the fused video features based on the Cauchy distribution mechanism to obtain distribution parameters corresponding to multiple video frames, wherein the distribution parameters represent the differences between the frame content of the video frame and the frame content of multiple other video frames in the video to be processed; a calculation module, configured to perform scoring calculation on the distribution parameters of the multiple video frames to obtain a scoring calculation result; an analysis module, configured to analyze the scoring calculation result based on a time uniform distribution condition to obtain multiple target video frames; and a second fusion module, configured to fuse the multiple target video frames to obtain a video thumbnail of the video to be processed.

[0014] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0016] A fifth aspect of the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] According to an embodiment of the present invention, attention fusion is performed on the initial video features of the video to be processed to obtain fused video features corresponding to multiple video frames of the video to be processed; the fused video features are processed based on the Cauchy distribution mechanism to obtain distribution parameters corresponding to multiple video frames; the distribution parameters of multiple video frames are scored to obtain scoring results; the scoring results are analyzed based on the time uniform distribution condition to obtain multiple target video frames; and the multiple target video frames are fused to obtain a video thumbnail of the video to be processed. Attention fusion of the initial video features results in fused video features with stronger multimodal adaptability and scalability, achieving better results in various video types and application scenarios. Processing the fused video features based on the Cauchy distribution mechanism can effectively identify parts of the video frames where information changes are concentrated, avoiding interference from irrelevant information. Attached Figure Description

[0018] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustrations depict application scenarios of the video thumbnail extraction method, apparatus, device, medium, and program products according to embodiments of the present invention.

[0020] Figure 2 A flowchart illustrating a video thumbnail extraction method according to an embodiment of the present invention is shown schematically.

[0021] Figure 3 The illustration shows a schematic diagram of a video thumbnail extraction method according to an embodiment of the present invention;

[0022] Figure 4 A schematic diagram illustrating the structure of a video thumbnail extraction apparatus according to an embodiment of the present invention is shown; and

[0023] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a video thumbnail extraction method according to an embodiment of the present invention. Detailed Implementation

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] In related technologies, video frames are uniformly extracted as thumbnails. This method usually does not take into account changes in the video content, which may result in the extracted thumbnails not fully reflecting the key information of the video.

[0029] For example, extracting color distribution, texture features, and edge information from each frame of an image can determine the importance of a video frame. This method can reflect the content information of the video to a certain extent, but it may be inaccurate in cases of complex backgrounds or high noise levels.

[0030] For example, deep learning models such as convolutional neural networks are used to classify video frames, and representative frames are selected based on the model's predictions. This type of method can handle complex image information to some extent; however, it typically requires a large amount of training data and computational resources, and is easily affected by noise and background changes, leading to unstable extraction results.

[0031] For example, temporal models such as Long Short-Term Memory (LSTM) networks can be used to capture the dynamic features and temporal dependencies of videos, thereby more accurately identifying key events or plots within them. However, these methods are computationally expensive and require high-quality training data.

[0032] This invention provides a video thumbnail extraction method, comprising: performing attention fusion on initial video features of the video to be processed to obtain fused video features corresponding to multiple video frames of the video to be processed, wherein the initial video features are obtained by feature extraction of the video to be processed, and the initial video features include initial image features, initial audio features, and initial text features; processing the fused video features based on the Cauchy distribution mechanism to obtain distribution parameters corresponding to multiple video frames, wherein the distribution parameters represent the differences between the frame content of the video frame and the frame content of multiple other video frames in the video to be processed; performing scoring calculation on the distribution parameters of the multiple video frames to obtain scoring calculation results; analyzing the scoring calculation results based on the time uniform distribution condition to obtain multiple target video frames; and fusing the multiple target video frames to obtain a video thumbnail of the video to be processed.

[0033] Figure 1 The illustration schematically depicts an application scenario of a video thumbnail extraction method, apparatus, device, medium, and program product according to embodiments of the present invention.

[0034] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the video thumbnail extraction method provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the video thumbnail extraction device provided in this embodiment of the invention can generally be located in server 105. The video thumbnail extraction method provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the video thumbnail extraction device provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] The following will be based on Figure 1 The described scenario provides a detailed description of the video thumbnail extraction method according to an embodiment of the present invention.

[0041] Figure 2 A flowchart illustrating a video thumbnail extraction method according to an embodiment of the present invention is shown.

[0042] like Figure 2 As shown, the video thumbnail extraction method of this embodiment includes operations S210 to S250.

[0043] In operation S210, attention fusion is performed on the initial video features of the video to be processed to obtain the fused video features corresponding to each of the multiple video frames of the video to be processed. The initial video features are obtained by feature extraction of the video to be processed, and the initial video features include initial image features, initial audio features and initial text features.

[0044] According to an embodiment of the present invention, the video to be processed may be a video captured at different scales for the same target object.

[0045] According to an embodiment of the present invention, the initial video features include initial image features, initial audio features, and initial text features. The initial image features may be the color, texture, and location of the target object in each frame of the video to be processed. The initial audio features may be audio signals such as the spectrum and amplitude generated by each frame of the video to be processed. The initial text features may be text descriptions configured for each frame of the video to be processed, such as subtitles.

[0046] According to embodiments of the present invention, attention fusion is performed on initial image features, initial audio features, and initial text features. The importance of different initial video features can be determined according to different attention weights, thereby generating fused video features corresponding to each video frame, so as to fully extract the key information of each frame image and facilitate the extraction of key frames.

[0047] In operation S220, the features of the fused video are processed based on the Cauchy distribution mechanism to obtain the distribution parameters corresponding to each of the multiple video frames. The distribution parameters represent the differences between the frame content of the video frame and the frame content of multiple other video frames in the video to be processed.

[0048] According to an embodiment of the present invention, the Cauchy distribution mechanism is a continuous probability distribution mechanism that can analyze the distribution of each video frame's fused video features among all fused video features, thereby obtaining the differences in frame content between each video frame and multiple other video frames in the video to be processed, so as to select representative video frames.

[0049] According to an embodiment of the present invention, the distribution parameters include location parameters and scale parameters, which are used to reflect the content importance of the fused video features among all fused video features, such as whether the content is sufficient, or whether it is extreme data.

[0050] In operation S230, the distribution parameters of multiple video frames are scored and calculated to obtain the score calculation results.

[0051] According to an embodiment of the present invention, the distribution parameters of multiple video frames are scored, and multiple video frames can be scored based on scale parameters. Furthermore, extreme data can be removed based on position parameters to obtain accurate scoring results.

[0052] In operation S240, the scoring calculation results are analyzed based on the time uniform distribution condition to obtain multiple target video frames.

[0053] According to an embodiment of the present invention, the time uniform distribution condition can ensure that there is a video frame that can be used as the target video frame in each time period, thereby avoiding oversampling of a certain time period.

[0054] The S250 is used to fuse multiple target video frames to obtain a video thumbnail of the video to be processed.

[0055] According to an embodiment of the present invention, multiple target video frames are superimposed and fused into a single image to obtain a video thumbnail of the video to be processed.

[0056] According to an embodiment of the present invention, attention fusion is performed on the initial video features of the video to be processed to obtain fused video features corresponding to multiple video frames of the video to be processed; the fused video features are processed based on the Cauchy distribution mechanism to obtain distribution parameters corresponding to multiple video frames; the distribution parameters of multiple video frames are scored to obtain scoring results; the scoring results are analyzed based on the time uniform distribution condition to obtain multiple target video frames; and the multiple target video frames are fused to obtain a video thumbnail of the video to be processed. Attention fusion of the initial video features results in fused video features with stronger multimodal adaptability and scalability, achieving better results in various video types and application scenarios. Processing the fused video features based on the Cauchy distribution mechanism can effectively identify parts of the video frames where information changes are concentrated, avoiding interference from irrelevant information.

[0057] According to an embodiment of the present invention, the distribution parameters include position parameters and scale parameters. The position parameters represent the importance of the frame content of the video frame, and the scale parameters represent the information density of the frame content of the video frame. A scoring calculation is performed on the distribution parameters of multiple video frames to obtain a scoring result, including: for any one of the multiple video frames, determining a score value based on the reciprocal of the scale parameter; selecting the distribution parameters of the multiple video frames that satisfy a preset distribution parameter threshold as target distribution parameters, and selecting the video frame corresponding to the target distribution parameter as a first candidate video frame; and determining the scoring result based on the score value corresponding to the first candidate video frame.

[0058] According to an embodiment of the present invention, It is a position parameter that indicates the center position of the video frame and reflects the core information of the video frame content. It is a scale parameter used to control the width of the distribution, representing the range of change of the video frame. The smaller the scale, the more concentrated the change of the video frame and the denser the information.

[0059] According to an embodiment of the present invention, the score value is determined based on the reciprocal of the scale parameter, as shown in formula (1).

[0060] (1).

[0061] in, As a rating value, This is the scale parameter.

[0062] According to embodiments of the present invention, a higher score indicates that the feature distribution of the video frame is more concentrated, the information density is higher, and the representativeness is stronger. Conversely, a lower score indicates that the video frame content is more redundant and the information density is lower.

[0063] According to an embodiment of the present invention, the distribution parameters of each of the multiple video frames are compared with preset distribution parameters to determine the difference between the distribution parameters of each video frame and the preset distribution parameter value. If the difference is less than the preset difference, the distribution parameters of the corresponding video frame are the target distribution parameters, and the video frame corresponding to the target distribution parameters is taken as the first candidate video frame.

[0064] According to an embodiment of the present invention, the score value corresponding to each of the first candidate video frames is used as the score calculation result.

[0065] According to an embodiment of the present invention, a score value is calculated for each frame, so that key frames with strong representativeness in the video can be sorted according to the score, ensuring that the selected first candidate video frame can retain the key information of the video content to the greatest extent.

[0066] According to embodiments of the present invention, the Cauchy distribution can effectively capture the changing trends between frames, especially the changes in important regions within a video frame, and reflects the "importance" and information density of the video frame in the video through positional and scale parameters.

[0067] According to an embodiment of the present invention, the video frame includes time period information; the scoring calculation results are analyzed based on the time uniform distribution condition to obtain multiple target video frames, including: obtaining a second candidate video frame based on the score value corresponding to the first candidate video frame according to the preset screening conditions; and selecting at least one second candidate video frame as a target video frame in each preset time period according to the time attributes of the multiple second candidate video frames.

[0068] According to an embodiment of the present invention, a scoring threshold is set. All first candidate video frames with scores greater than the threshold are considered as second candidate video frames.

[0069] According to an embodiment of the present invention, in order to ensure that the selected target video frames can cover the entire timeline of the video to be processed, the target video frames are selected in a uniformly distributed manner at intervals, ensuring that at least one second candidate video frame can be selected as the target video frame in each time period. The time attribute of each second candidate video frame can be the time period to which each second candidate video frame belongs on the timeline.

[0070] According to an embodiment of the present invention, considering the uniformity of the time axis, it is ensured that the selected target video frames can cover the main content of the video, thus avoiding the problem of oversampling of a certain part of the frames.

[0071] According to an embodiment of the present invention, attention fusion is performed on video features to obtain fused video features for each of the multiple video frames of the video to be processed, including: for any one of the multiple video frames, weighted fusion is performed on the initial image features, initial audio features and initial text features at different scales according to their respective attention weights to obtain the fused video features of the video frame.

[0072] According to embodiments of the present invention, the initial image features at different scales can be image features extracted at different spatial resolutions, the initial audio features at different scales can be audio features extracted at different frequency ranges or time window sizes, and the initial text features at different scales can be text features extracted at different text lengths or context ranges.

[0073] According to an embodiment of the present invention, the initial image features, initial audio features, and initial text features are weighted and fused at different scales using their respective attention weights. Specifically, the initial image features at different scales may be fused first, for example, by splicing, to obtain fused initial image features. Then, the fused initial image features, fused initial audio features, and fused initial text features are weighted and fused using their respective attention weights to obtain fused video features of the video frame.

[0074] According to embodiments of the present invention, initial image features, initial audio features, and initial text features at different scales are weighted and fused to make the fused video features more adaptable to multiple modalities and scalable, and to achieve better results in various video types and application scenarios.

[0075] According to an embodiment of the present invention, the above method further includes: performing attention fusion on sample video features based on attention weights to obtain sample fused video features for each of multiple sample video frames of the sample video, wherein the sample video features are obtained by feature extraction of the sample video; processing the sample fused video features based on the Cauchy distribution mechanism to obtain sample distribution parameters for each of the multiple sample video frames; calculating a score for each of the sample distribution parameters of the multiple sample video frames to obtain a sample score calculation result; analyzing the sample score calculation result based on the time uniform distribution condition to obtain multiple target sample video frames; optimizing the attention weights according to the loss value between the target sample video frames and the label video frames to obtain optimized attention weights; wherein, weighted fusion of initial image features, initial audio features, and initial text features at different scales is performed based on the attention weights of the initial image features, initial audio features, and initial text features, including: weighted fusion of initial image features, initial audio features, and initial text features at different scales based on the optimized attention weights of the initial image features, initial audio features, and initial text features.

[0076] According to an embodiment of the present invention, when training the optimized attention weights of the initial image features, initial audio features, and initial text features, the difference between the target sample video frame and the label video frame is used as the loss value. Based on this loss value, the attention weights can be iteratively optimized until the loss value meets the preset loss value, thereby completing the optimization of the attention weights.

[0077] According to an embodiment of the present invention, the fused video features are processed based on the Cauchy distribution mechanism to obtain the distribution parameters corresponding to each of the multiple video frames, including: updating the probability density function of the Cauchy distribution based on the fused video features to obtain the likelihood function; performing a natural logarithm calculation operation on the likelihood function to obtain the target likelihood function; and solving the target likelihood function based on a preset solution objective to obtain the distribution parameters, wherein the preset solution objective indicates that the solution value of the target likelihood function satisfies a preset solution condition.

[0078] According to an embodiment of the present invention, for each video frame, the fused video features The Cauchy distribution is used to model it, and the probability density function of the Cauchy distribution is shown in formula (2).

[0079] (2).

[0080] in, It is a position parameter that indicates the center position of the video frame and reflects the core information of the video frame content. The scale parameter controls the width of the video frame distribution, representing the amplitude of change within that video frame. A smaller scale indicates more concentrated change and denser information within the video frame. The Cauchy distribution effectively captures the changing trends between video frames, especially changes in important regions within a video frame. The Cauchy distribution parameters for each video frame are obtained by fitting the view fusion video features. and These two parameters reflect the "importance" and information density of the video frame in the video.

[0081] According to embodiments of the present invention, when utilizing fused video features After updating formula (2) above, the above problem can be solved by taking the logarithm. and In order to obtain the distribution parameters.

[0082] According to an embodiment of the present invention, during the solution process, it is necessary to make the function value of the target likelihood function reach a preset function threshold, or after completing a specified number of iterations, select the parameter corresponding to the maximum function value as the distribution parameter to be obtained.

[0083] According to embodiments of the present invention, the peak characteristics of the Cauchy distribution can effectively identify key information in video frames, avoiding redundant information and inefficient extraction problems caused by uniform sampling or simple statistical methods in related technologies. The Cauchy distribution has strong peak characteristics, making it suitable for capturing extreme values ​​and outliers in data, thus enabling more accurate modeling of frame features.

[0084] According to an embodiment of the present invention, fusing multiple target video frames to obtain a video thumbnail of the video to be processed includes: stitching multiple target video frames together to obtain an initial video thumbnail; and performing image enhancement on the initial video thumbnail to obtain a video thumbnail of the video to be processed.

[0085] According to an embodiment of the present invention, multiple target video frames are merged into a single thumbnail using a stitching method to obtain an initial video thumbnail.

[0086] According to embodiments of the present invention, the synthesized initial video thumbnail is post-optimized using image processing techniques, including: image enhancement, such as adjusting image brightness, image contrast, and image sharpness to improve the visual effect of the image; noise removal, such as using filtering techniques to remove redundant noise from the image to ensure image clarity; and background processing, such as removing unnecessary image background information to highlight the core content of the video.

[0087] According to embodiments of the present invention, the generated video thumbnails display video content from multiple angles through multi-frame synthesis. Furthermore, post-processing optimizations, such as image enhancement and noise removal, are performed to ensure the generated video thumbnails have good visual effects.

[0088] According to an embodiment of the present invention, feature extraction of the video to be processed includes: sampling the video to be processed to obtain a specified number of video frames per second; extracting features from the color values ​​of pixels in the video frames, texture information sequences associated with the video frames, and position information of the target object based on a local binary extraction algorithm to obtain initial image features, wherein the texture information sequence represents the arrangement result of pixel texture information of multiple pixels; extracting features from the audio signals of the video frames based on a frequency domain extraction algorithm to obtain initial audio features; and extracting features from the text information of the video frames based on a text recognition algorithm to obtain initial text features.

[0089] According to an embodiment of the present invention, the video is uniformly sampled according to the video frame rate, and N frames are selected from the video for analysis every second. Assuming the video length is T seconds, a total of N×T frames are sampled from the video.

[0090] According to embodiments of the present invention, since the extracted features include initial image features, initial audio features, and initial text features, they can be adapted to different types of videos (such as short videos, movies, educational videos, etc.) and can be widely used in fields such as video indexing, video recommendation, and multimedia information retrieval.

[0091] Figure 3 The illustration shows a schematic diagram of a video thumbnail extraction method according to an embodiment of the present invention.

[0092] According to embodiments of the present invention, such as Figure 3 As shown, in operation S301, video frames are sampled and features are extracted from the video to be processed to obtain initial video features; in operation S302, attention fusion is performed on the initial video features to obtain fused video features; in operations S303 and S304, the fused video features are processed based on the Cauchy distribution mechanism to obtain position parameters and scale parameters, respectively; in operation S305, the position parameters and scale parameters are calculated to obtain the score calculation result; in operation S306, the score value in the score calculation result is judged according to the preset screening conditions. If the video frame does not meet the preset screening conditions, operation S307 is executed to discard it. If it meets the conditions, operation S308 is executed to set it as the target video frame, and operation S309 is executed to stitch multiple target video frames together. Then, in operation S310, an initial video thumbnail is generated; further, operation S311 is executed to perform image enhancement, noise removal, and background processing on the initial video thumbnail, and in operation S312, the video thumbnail is output.

[0093] Based on the above-described video thumbnail extraction method, this invention also provides a video thumbnail extraction device. The following will combine... Figure 4 The video thumbnail extraction device is described in detail.

[0094] Figure 4 A schematic block diagram of a video thumbnail extraction device according to an embodiment of the present invention is shown.

[0095] like Figure 4 As shown, the video thumbnail extraction device 400 of this embodiment includes a first fusion module 410, a processing module 420, a calculation module 430, an analysis module 440, and a second fusion module 450.

[0096] The first fusion module 410 is used to perform attention fusion on the initial video features of the video to be processed, thereby obtaining fused video features corresponding to each of the multiple video frames of the video to be processed. The initial video features are obtained by feature extraction from the video to be processed, and include initial image features, initial audio features, and initial text features. In one embodiment, the first fusion module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0097] The processing module 420 processes the fused video features based on the Cauchy distribution mechanism to obtain distribution parameters corresponding to each of the multiple video frames. These distribution parameters represent the differences between the frame content of one video frame and the frame content of multiple other video frames in the video to be processed. In one embodiment, the generation module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0098] The calculation module 430 is used to calculate the scores of the distribution parameters of multiple video frames to obtain the score calculation results. In one embodiment, the calculation module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0099] The analysis module 440 is used to analyze the scoring calculation results based on the time uniform distribution condition to obtain multiple target video frames. In one embodiment, the analysis module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0100] The second fusion module 450 is used to fuse multiple target video frames to obtain a video thumbnail of the video to be processed. In one embodiment, the second fusion module 450 can be used to perform the operation S250 described above, which will not be repeated here.

[0101] According to an embodiment of the present invention, attention fusion is performed on the initial video features of the video to be processed to obtain fused video features corresponding to multiple video frames of the video to be processed; the fused video features are processed based on the Cauchy distribution mechanism to obtain distribution parameters corresponding to multiple video frames; the distribution parameters of multiple video frames are scored to obtain scoring results; the scoring results are analyzed based on the time uniform distribution condition to obtain multiple target video frames; and the multiple target video frames are fused to obtain a video thumbnail of the video to be processed. Attention fusion of the initial video features results in fused video features with stronger multimodal adaptability and scalability, achieving better results in various video types and application scenarios. Processing the fused video features based on the Cauchy distribution mechanism can effectively identify parts of the video frames where information changes are concentrated, avoiding interference from irrelevant information.

[0102] According to embodiments of the present invention, any plurality of modules among the first fusion module 410, processing module 420, computing module 430, analysis module 440, and second fusion module 450 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the first fusion module 410, processing module 420, computing module 430, analysis module 440, and second fusion module 450 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the first fusion module 410, processing module 420, calculation module 430, analysis module 440, and second fusion module 450 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0103] According to an embodiment of the present invention, the distribution parameters include position parameters and scale parameters. The position parameters represent the importance of the frame content of the video frame, and the scale parameters represent the information density of the frame content of the video frame. The calculation module 430 includes a first determination submodule, a first use submodule, and a second determination submodule.

[0104] The first determination submodule is used to determine the score value for any one of the multiple video frames based on the reciprocal of the scale parameter.

[0105] The first module is used as a submodule to take the distribution parameters of multiple video frames that meet the preset distribution parameter threshold as the target distribution parameter, and take the video frame corresponding to the target distribution parameter as the first candidate video frame.

[0106] The second determining submodule is used to determine the score calculation result based on the score value corresponding to the first candidate video frame.

[0107] According to an embodiment of the present invention, the video frame includes time period information; the analysis module 440 includes a first obtaining submodule and a first filtering submodule.

[0108] The first submodule is used to obtain the second candidate video frame based on the score value corresponding to the first candidate video frame according to preset filtering conditions.

[0109] The first filtering submodule is used to select at least one second candidate video frame as the target video frame in each preset time period based on the time attributes of the multiple second candidate video frames.

[0110] According to an embodiment of the present invention, the first fusion module 410 includes a first fusion submodule.

[0111] The first fusion submodule is used to perform weighted fusion of the initial image features, initial audio features, and initial text features at different scales for any one of the multiple video frames, based on the attention weights of the initial image features, initial audio features, and initial text features, to obtain the fused video features of the video frame.

[0112] According to an embodiment of the present invention, the video thumbnail extraction device 400 further includes a first obtaining module, a second obtaining module, a third obtaining module, a fourth obtaining module, and a fifth obtaining module; the first fusion module 410 further includes a second fusion submodule.

[0113] The first module is used to perform attention fusion on the sample video features based on attention weights to obtain sample fused video features for each of the multiple sample video frames of the sample video, wherein the sample video features are obtained by feature extraction of the sample video.

[0114] The second module is used to process the sample fusion video features based on the Cauchy distribution mechanism to obtain the sample distribution parameters of each of the multiple sample video frames.

[0115] The third module is used to calculate the score of the sample distribution parameters of multiple sample video frames and obtain the sample score calculation result.

[0116] The fourth module is used to analyze the sample scoring calculation results based on the time uniform distribution condition, and obtain multiple target sample video frames.

[0117] The fifth module is used to optimize the attention weights based on the loss value between the target sample video frame and the label video frame, and obtain the optimized attention weights.

[0118] The second fusion submodule is used to perform weighted fusion of the initial image features, initial audio features, and initial text features at different scales based on the optimized attention weights of the initial image features, initial audio features, and initial text features respectively.

[0119] According to an embodiment of the present invention, the processing module 420 includes a first update submodule, a second obtain submodule, and a first solver submodule.

[0120] The first update submodule is used to update the probability density function of the Cauchy distribution based on the fused video features to obtain the likelihood function.

[0121] The second submodule is used to perform the natural logarithm calculation on the likelihood function to obtain the target likelihood function.

[0122] The first solution submodule is used to solve the target likelihood function based on a preset solution objective to obtain the distribution parameters. The preset solution objective indicates that the solution value of the target likelihood function satisfies the preset solution conditions.

[0123] According to an embodiment of the present invention, the second fusion submodule includes a splicing submodule and an enhancement submodule.

[0124] The stitching submodule is used to stitch together multiple target video frames to obtain an initial video thumbnail.

[0125] The enhancement submodule is used to enhance the initial video thumbnail to obtain a video thumbnail of the video to be processed.

[0126] According to an embodiment of the present invention, the first fusion module 410 further includes a sampling submodule, a first extraction submodule, a second extraction submodule, and a third extraction submodule.

[0127] The sampling submodule is used to sample the video to be processed, obtaining a specified number of video frames per second.

[0128] The first extraction submodule is used to extract features from the color information, texture information and position information of the target object in the video frame based on the local binary extraction algorithm, so as to obtain the initial image features.

[0129] The second extraction submodule is used to extract features from the audio signal of the video frame based on the frequency domain extraction algorithm to obtain the initial audio features.

[0130] The third extraction submodule is used to extract features from the text information of video frames based on text recognition algorithms to obtain initial text features.

[0131] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a video thumbnail extraction method according to an embodiment of the present invention.

[0132] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0133] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0134] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0135] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0136] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0137] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the video thumbnail extraction method provided in the embodiments of the present invention.

[0138] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0139] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0140] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0141] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0143] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0144] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for extracting video thumbnails, characterized in that, The method includes: Attention fusion is performed on the initial video features of the video to be processed to obtain fused video features corresponding to each of the multiple video frames of the video to be processed. The initial video features are obtained by feature extraction of the video to be processed, and the initial video features include initial image features, initial audio features and initial text features. The fused video features are processed based on the Cauchy distribution mechanism to obtain distribution parameters corresponding to multiple video frames. The distribution parameters represent the differences between the frame content of the video frame and the frame content of multiple other video frames in the video to be processed. The distribution parameters of multiple video frames are scored to obtain the scoring results. The scoring calculation results are analyzed based on the time uniform distribution condition to obtain multiple target video frames; By fusing multiple target video frames, a video thumbnail of the video to be processed is obtained.

2. The method according to claim 1, characterized in that, The distribution parameters include position parameters and scale parameters. The position parameters represent the importance of the frame content of the video frame, and the scale parameters represent the information density of the frame content of the video frame. The scoring calculation of the distribution parameters of multiple video frames to obtain the scoring result includes: For any one of the multiple video frames, the score is determined based on the reciprocal of the scale parameter; The distribution parameters that satisfy the preset distribution parameter threshold among the distribution parameters of multiple video frames are taken as the target distribution parameters, and the video frames corresponding to the target parameters are taken as the first candidate video frames. The scoring result is determined based on the score value corresponding to the first candidate video frame.

3. The method according to claim 2, characterized in that, The video frames include time period information; The scoring calculation results are analyzed based on the time uniform distribution condition to obtain multiple target video frames, including: Based on preset filtering conditions, a second candidate video frame is obtained according to the score value corresponding to the first candidate video frame; Based on the time attributes of each of the multiple second candidate video frames, at least one second candidate video frame is selected as the target video frame in each preset time period.

4. The method according to claim 1, characterized in that, The attention fusion of video features to obtain fused video features for each of the multiple video frames of the video to be processed includes: For any one of the multiple video frames, the initial image features, initial audio features, and initial text features at different scales are weighted and fused according to their respective attention weights to obtain the fused video features of the video frame.

5. The method according to claim 4, characterized in that, The method further includes: Based on the attention weights, attention fusion is performed on the sample video features to obtain sample fused video features for each of the multiple sample video frames of the sample video, wherein the sample video features are obtained by feature extraction of the sample video; The sample fusion video features are processed based on the Cauchy distribution mechanism to obtain the sample distribution parameters of each of the multiple sample video frames; The sample distribution parameters of multiple sample video frames are scored to obtain the sample score calculation results. The sample scoring calculation results are analyzed based on the time uniform distribution condition to obtain multiple target sample video frames; The attention weights are optimized based on the loss value between the target sample video frame and the label video frame to obtain the optimized attention weights. The step of weighted fusion of the initial image features, initial audio features, and initial text features at different scales based on their respective attention weights includes: Based on the optimized attention weights of the initial image features, the initial audio features, and the initial text features, the initial image features, the initial audio features, and the initial text features at different scales are weighted and fused.

6. The method according to claim 1, characterized in that, The fused video features are processed based on the Cauchy distribution mechanism to obtain distribution parameters corresponding to each of the multiple video frames, including: The probability density function of the Cauchy distribution is updated based on the fused video features to obtain the likelihood function; Perform the natural logarithm calculation on the likelihood function to obtain the target likelihood function; Based on a preset solution objective, the objective likelihood function is solved to obtain the distribution parameters, wherein the preset solution objective indicates that the solution value of the objective likelihood function satisfies a preset solution condition.

7. The method according to claim 1, characterized in that, The process of fusing multiple target video frames to obtain a video thumbnail of the video to be processed includes: Multiple target video frames are stitched together to obtain an initial video thumbnail; Image enhancement is performed on the initial video thumbnail to obtain the video thumbnail of the video to be processed.

8. The method according to claim 1, characterized in that, The feature extraction of the video to be processed includes: The video to be processed is sampled to obtain a specified number of video frames per second. Based on the local binary extraction algorithm, the color values ​​of pixels in the video frame, the texture information sequence associated with the video frame, and the position information of the target object are extracted to obtain the initial image features. The texture information sequence represents the arrangement result of the pixel texture information of multiple pixels. Based on the frequency domain extraction algorithm, feature extraction is performed on the audio signal of the video frame to obtain the initial audio features; Based on a text recognition algorithm, feature extraction is performed on the text information of the video frame to obtain the initial text features.

9. A video thumbnail extraction device, characterized in that, The device includes: The first fusion module is used to perform attention fusion on the initial video features of the video to be processed to obtain fused video features corresponding to each of the multiple video frames of the video to be processed. The initial video features are obtained by feature extraction of the video to be processed, and the initial video features include initial image features, initial audio features and initial text features. The processing module is used to process the fused video features based on the Cauchy distribution mechanism to obtain the distribution parameters corresponding to each of the multiple video frames. The distribution parameters represent the differences between the frame content of the video frame and the frame content of multiple other video frames in the video to be processed. The calculation module is used to calculate the score of the distribution parameters of multiple video frames and obtain the score calculation result. The analysis module is used to analyze the scoring calculation results based on the time uniform distribution condition to obtain multiple target video frames; The second fusion module is used to fuse multiple target video frames to obtain a video thumbnail of the video to be processed.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

Images