Video key frame extraction method and device applied to clothing special place live video intelligent slice

The method scores and selects key frames in live clothing show videos based on frame characteristics and product matching, addressing resource waste and inaccuracy in existing methods by integrating perspective, display form, and target quality, resulting in improved segmentation.

CN119094803APending Publication Date: 2024-12-06XIAMEN CICADA NETWORK TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411083267.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing video frame extraction methods for live clothing shows waste computing resources and lead to inaccurate segmentation due to invalid frames, mislead final results, and reduce actual sampling frequency.

Method used

A method and device for intelligent slicing of live clothing show videos that scores and selects key frames based on the video frame's characteristics and product matching, integrating perspective, display form, and target quality, using multimodal feature extraction to identify relevant frames.

Benefits of technology

Accurately identifies key frames that are relevant to the task, reducing processing of invalid frames and improving computing resource utilization by ensuring frames are closely related to the product, thus enhancing the segmentation effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094803A_ABST
    Figure CN119094803A_ABST
Patent Text Reader

Abstract

The invention provides a video key frame extraction method and device applied to clothing special place live video intelligent slicing, and relates to the technical field of video key frame extraction, and the method comprises the steps: obtaining a live video, an explanation document and a commodity set, and carrying out the framing processing of the live video, and generating a video frame set; key frame screening processing is carried out on the video frame set, the explanation copywriting and the commodity set, a frame extraction list is generated, scoring processing is carried out on each frame of picture in the video frame set, key frames are selected according to scores and frame extraction frequency, and scoring processing comprises score scoring processing of the video frames and scoring processing of matching scores of the video frames and commodities. The key frame is accurately identified by combining the picture quality of the video frame, the information of the commodities in the video and the matching degree information of the video frame and the target commodity set.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video key frame extraction, and in particular to a video key frame extraction method and device applied to intelligent slicing of live video of a clothing show. Background Art

[0002] The task of intelligent slicing of live video refers to automatically slicing video clips related to products, such as product introduction, Q&A and display, from the recorded video in the live broadcast room, and giving the products involved in the clip. Video processing is generally converted into an image processing task for video frame processing, but there are many video frames per unit time, and the computing resources for frame-by-frame processing are usually unacceptable. However, the similarity of similar frames is extremely high, especially in this scenario, similar frames generally explain the same product, so it is extremely important to screen the video frames in the task and identify the key frames, and the quality of the key frames directly affects the final effect of the entire task. At present, frame extraction technology generally uses uniform sampling to extract frames, such as extracting a frame every certain period of time or every certain number of frames.

[0003] The uniform frame extraction method currently used in the market may extract invalid frames, such as poor picture quality caused by excessive motion, no products in the picture, no target products to be segmented in the picture, etc. This may reduce the actual sampling frequency due to more invalid frames (for example, if a frame is extracted every 10 seconds, if there are two consecutive invalid frames, the interval between them will be 30 seconds), waste computing resources (invalid frames will also occupy computing resources in subsequent judgment processing), and even mislead the final segmentation results.

[0004] In view of this, this application is filed. Summary of the invention

[0005] The present invention provides a method and device for extracting video key frames for intelligent slicing of live video of a clothing show, which can at least partially improve the above-mentioned problems.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A video key frame extraction method for intelligent slicing of clothing live broadcast video, comprising:

[0008] Acquire a live video, explanation copy, and a product set, and perform frame processing on the live video to generate a video frame set;

[0009] The video frame set, explanation copy and product set are subjected to key frame screening processing to generate a frame extraction list, wherein each frame of the video frame set is scored and key frames are selected according to the score and the frame extraction frequency.

[0010] The present invention also provides a video key frame extraction device for intelligent slicing of clothing live broadcast video, which includes:

[0011] A frame division unit is used to obtain a live video, an explanation copy and a product set, and perform frame processing on the live video to generate a video frame set;

[0012] The key frame screening unit is used to perform key frame screening processing on the video frame set, explanation copy and product set to generate a frame extraction list, wherein each frame of the video frame set is scored, and key frames are selected according to the score and the frame extraction frequency. The scoring process includes the score scoring process of the video frame itself and the score scoring process of matching with the product.

[0013] In summary, the video key frame extraction method applied to intelligent slicing of clothing live broadcast videos first divides the live broadcast video into frames, and then performs key frame screening based on the video's explanation text and product collection. Specifically, in the intelligent slicing of the live video of the clothing show, the method calculates the importance of the video frame according to the characteristics of the video frame itself and the matching degree between the video and the product to be segmented; in the calculation of the importance of the video frame, the product targets in the video frame and the product image are identified to reduce the interference of background information; in the calculation of the importance of the video itself, the three factors of perspective, display form and target quality are integrated, and some video frames are filtered out of interference data by target quality; in the calculation of the product matching score, the multimodal information of the product target picture and the text image of the explanation copy and the product title is considered; in the calculation of the product matching score, the product target in the video frame is matched with the product targets in the list of all products to be segmented, and the maximum matching value is taken as the product matching score of the video frame; in the process of selecting the key frame set, the video is firstly segmented into equal lengths, and then the frames in the segment are extracted. Compared with directly selecting according to the importance, it is ensured that the interval between the two adjacent frames selected is less than twice the frame extraction frequency. In short, the video keyframe extraction method applied to intelligent slicing of clothing live broadcast videos accurately identifies keyframes by combining the image quality of video frames, information about products in the video, and matching information between video frames and target product sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of a method for extracting video key frames for intelligent slicing of live video of a clothing show provided by the first aspect of the present invention;

[0015] Figure 2 It is a flow chart of a video key frame extraction method for intelligent slicing of live video of a clothing show provided by the second aspect of the present invention;

[0016] Figure 31 is a schematic diagram of a key frame screening process of a video key frame extraction method for intelligent slicing of a clothing live broadcast video provided by the first embodiment of the present invention;

[0017] Figure 4 It is a module diagram of a video key frame extraction device for intelligent slicing of live clothing show videos provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] refer to Figure 1 , Figure 2 As shown, the first embodiment of the present invention discloses a video key frame extraction method applied to intelligent slicing of clothing live broadcast video, which can be executed by a video key frame extraction device applied to intelligent slicing of clothing live broadcast video (hereinafter referred to as extraction device), and in particular, executed by one or more processors in the extraction device to implement the following method:

[0020] S101, obtaining a live video, an explanation copy, and a product set, and performing frame processing on the live video to generate a video frame set;

[0021] S102, performing key frame screening processing on the video frame set, explanation copy and product set to generate a frame extraction list, wherein each frame in the video frame set is scored, and key frames are selected according to the score and the frame extraction frequency. The scoring process includes the scoring process of the video frame itself and the scoring process of the matching score with the product.

[0022] Preferably, the score of the video frame itself includes a viewing angle score, a display form score and a target quality score.

[0023] Specifically, step S102 includes: performing commodity detection processing on the video frame set and the commodity images of the commodity set to generate a video frame commodity target set and a commodity image commodity target set;

[0024] Respectively performing viewpoint recognition, display form recognition, and target quality recognition on the set of commodity targets in the video frame to obtain a viewpoint score, a display form score, and a target quality score;

[0025] Input the explanation copy, the video frame product target set, the product image product target set and the product titles in the product set into a multimodal feature extraction model for feature extraction processing, obtain the product image product target feature code and the video frame product target feature code, and calculate the product image product target feature code and the video frame product target feature code to generate a product matching score;

[0026] The viewing angle score, display format score, target quality score and product matching score are added together to obtain a video frame importance score, and key frame screening is performed in combination with the frame extraction frequency to obtain a key frame set;

[0027] A frame extraction list is generated according to the key frame set.

[0028] See also Figure 3 In this embodiment, each frame of the picture is scored, and then the key frame is selected according to the score and the frame extraction frequency. The score includes the score of the video frame itself and the matching score with the product. The score of the video frame itself includes the view angle, display form, and target quality score. First, the video frame is detected for the product to obtain the set of product targets corresponding to the video frame, and then the view angle, display form, and target quality of the product in the video frame are scored. Then, combined with the multimodal feature extraction model, the video frame product target and the video explanation copy are feature extracted to obtain the feature vector of the video frame, and then the product image product target and product title information are feature extracted to obtain the feature vector of the product. The product matching score corresponding to the video frame is calculated by the feature vector of the video frame and the product. Finally, the view angle, display form, target quality and product matching score of the video frame are integrated to obtain the final importance score of the video frame, and then the key frame is screened according to the importance score.

[0029] Preferably, the video frame commodity target set is subjected to perspective recognition to obtain a perspective score, specifically:

[0030] A classification neural network is used for perspective recognition, wherein the perspective categories include front, back, left and right, and the perspective score is the predicted probability of the front category;

[0031] When it is determined that multiple commodity targets are extracted from a frame of video, the highest score of all target perspectives is taken as the perspective score of the frame. The calculation formula is:

[0032]

[0033] Among them, T f are all the targets identified in the f-th frame, p(t,c) is the predicted probability that target t belongs to category c, and S(f) is the view score of the f-th frame.

[0034] In this embodiment, perspective recognition is a classification neural network, and the categories include front, back, left and right. Because the front is usually the main view in live broadcasts and main product pictures, the perspective score is the predicted probability of the "front" category. If multiple product targets are extracted from a frame of video, the perspective score is the highest score of all target perspective scores as the perspective score of the frame.

[0035] Preferably, the display form of the commodity target set in the video frame is identified to obtain a display form score, specifically:

[0036] A classification neural network is used to identify the display form, wherein the display form categories include try-on and non-try-on, and the display form score is the predicted probability of the try-on category;

[0037] When it is determined that multiple commodity targets are extracted from a frame of video, the highest score among all target display forms is used as the display form score of the frame.

[0038] In this embodiment, display form recognition is also a classification neural network, and the categories include try-on and non-try-on. Because try-on is generally the main display form in live videos, the display form score is the predicted probability of try-on. If multiple product targets are extracted from a frame of video, the display form score is the highest score of all target display form scores as the display form score of the frame. The calculation logic is similar to the viewing angle score, which will not be repeated here.

[0039] Preferably, target quality recognition is performed on the video frame commodity target set to obtain a target quality score, specifically:

[0040] Using a binary classification neural network to identify target quality, wherein the target quality categories include good and bad, and the target quality score is the predicted probability of the good category;

[0041] When it is determined that multiple product targets are extracted from a frame of video, the lowest quality score of all targets is used as the target quality score of the frame;

[0042] When it is determined that the predicted probability of any product target quality being a bad category is greater than 0.5, the frame is discarded and does not enter the subsequent screening stage.

[0043] In this embodiment, target quality recognition is a binary classification neural network, the categories include good and bad, and the target quality score is the predicted probability of "good". If multiple commodity targets are extracted from a frame of video, the quality score is the lowest score of all target quality scores as the quality score of the frame. It should be noted that if the predicted probability of "bad" in any commodity target quality is greater than 0.5, the frame is discarded and does not enter the subsequent screening stage.

[0044] Preferably, the explanation copy, the video frame product target set, the product image product target set and the product title in the product set are input into a multimodal feature extraction model for feature extraction processing to obtain product image product target feature codes and video frame product target feature codes, and the product image product target feature codes and video frame product target feature codes are calculated to generate a product matching score, specifically:

[0045] Use a multimodal feature extraction model to perform feature encoding processing on each product target on the product side and the video frame side to obtain product target feature encoding of the product image and the video frame. The model input on the video frame side is the explanation copy within a 20-second time window 10 seconds before and after the corresponding time of the product target image and the video frame in the current video. The model input on the product side is the product target image on the product side and the title of the current product.

[0046] According to the commodity target feature coding of the commodity image and the commodity target feature coding of the video frame, the pairwise matching degree of all commodity targets on the video frame side and the commodity side is calculated, and the matching value with the highest matching degree is taken as the commodity matching score of the video frame.

[0047] Preferably, the product matching score represents a score of the matching degree between the video frame and the list of products to be segmented.

[0048] In this embodiment, first, feature encoding is performed on each product target on the product side and the video frame side through a multimodal feature extraction model. The model input on the video frame side is the product target picture of the current video and the explanation text within a 20-second time window 10 seconds before and after the corresponding time of the video frame; the model input on the product side is the product target picture on the product side and the title of the current product. After obtaining the feature encoding at both ends, the pairwise matching degree of all product targets on the video frame side and the product side is calculated. Because there may be multiple product targets in a video frame, there may be multiple product images for a product in the product list, and there may be multiple product targets for a product image, so the matching value with the highest matching degree is taken here as the product matching score of the video frame.

[0049] Preferably, the multimodal feature extraction model uses an open source multimodal model clip, wherein the matching degree measurement function uses cosine similarity.

[0050] In this embodiment, the multimodal feature extraction model can use an open source multimodal model, such as clip, etc., or it can be obtained by pre-training. Here, clip is taken as an example, and the matching degree measurement function uses cosine similarity. The specific calculation method is as follows:

[0051] For feature coding, for each product target t in each video frame f and each product target i of product g in each product list, their feature coding can be obtained through the feature extraction model. For the video frame, the formula is:

[0052]

[0053] For commodities, the formula is:

[0054]

[0055] Among them, context f It is the explanation text in the 20-second time window 10 seconds before and after the corresponding time of video frame f, title g This is the title of product g.

[0056] For similarity calculation, for each product target t in each video frame f and each product target i of product g in each product list, their similarity (matching degree) can be calculated, and the formula is:

[0057]

[0058] For product matching score calculation, for each video frame f, the highest matching degree is selected as the product matching score of the video frame. The formula is:

[0059]

[0060] Among them, T f is the set of all commodity targets identified in the video frame f, G is the commodity list, I g is the set of all commodity targets in commodity g.

[0061] Finally, the final score of the video frame is obtained by adding the product matching score of the video frame, the viewing angle, the display format, and the target quality score, and then the key frame is screened in combination with the frame extraction frequency threshold.

[0062] Given a frame extraction frequency r, the entire video is divided into n segments of r length each, and then the frame with the highest score is extracted from the segment as the key frame in the current window, and finally the frame extraction result of the entire video is obtained, so that the maximum interval between two adjacent frames is 2r.

[0063] In summary, the video keyframe extraction method applied to the intelligent slicing of clothing live broadcast videos proposes a keyframe screening scheme for intelligent slicing of clothing live broadcast videos that integrates the characteristics of the video frame itself and the matching degree between the video and the product to be segmented. Compared with uniform sampling, it can screen out keyframes that are more relevant to the task and improve the final segmentation effect. In addition, in the intelligent slicing of clothing live broadcast videos, the importance of the video frame is calculated based on the characteristics of the video frame itself and the matching degree between the video and the product to be segmented. Compared with the traditional keyframe screening method based on video frame features, more keyframes related to the product can be screened out, thereby improving the final segmentation effect, reducing the processing of invalid frames, and improving the utilization rate of subsequent computing resources.

[0064] In addition, in the calculation of the importance of the video itself, the three factors of perspective, display form, and target quality are integrated, and some video frames are filtered by target quality to eliminate interference data. The video frames are screened by target quality to reduce their impact on subsequent processing, and the three factors of perspective, display form, and target quality are integrated to effectively identify frames that are beneficial to the task of slicing the product explanation time period. In the calculation of product matching scores, the multimodal information of the product target picture and the text image of the explanation copy and product title is considered. On the basis of the picture, more text multimodal information can be considered to increase the recognition effect of key frames. In the calculation of product matching scores, the product target in the video frame is matched with the product target in the list of all products to be segmented, and the maximum matching value is taken as the product matching score of the video frame, which reduces the interference of irrelevant products. In the process of selecting the key frame set, the video is first divided into equal lengths, and then the frames in the segment are extracted. Compared with directly selecting according to importance, it ensures that the interval between the selected two adjacent frames is less than twice the frame extraction frequency.

[0065] See also Figure 4 The second embodiment of the present invention further provides a video key frame extraction device for intelligent slicing of clothing live broadcast video, which includes:

[0066] The frame division unit 201 is used to obtain the live video, the explanation copy and the product set, and perform frame division processing on the live video to generate a video frame set;

[0067] The key frame screening unit 202 is used to perform key frame screening processing on the video frame set, explanation copy and product set to generate a frame extraction list, wherein each frame of the video frame set is scored, and key frames are selected according to the score and the frame extraction frequency. The scoring process includes the scoring process of the video frame itself and the scoring process of the matching score with the product.

[0068] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A video key frame extraction method for intelligent slicing of live video of clothing shows, characterized in that: include: Acquire a live video, explanation copy, and a product set, and perform frame processing on the live video to generate a video frame set; The video frame set, explanation text and product set are subjected to key frame screening processing to generate a frame extraction list, wherein each frame in the video frame set is scored, and key frames are selected according to the scores and the frame extraction frequency, and the scoring processing includes the scoring processing of the video frame itself and the scoring processing of the matching score with the product.

2. The video key frame extraction method for intelligent slicing of clothing live broadcast video according to claim 1 is characterized in that: The scores of the video frames themselves include viewing angle score, presentation form score and target quality score.

3. The video key frame extraction method for intelligent slicing of clothing live broadcast video according to claim 1 is characterized in that: The video frame set, explanation copy and product set are subjected to key frame screening processing to generate a frame extraction list, specifically: Performing commodity detection processing on the commodity images of the video frame set and the commodity set to generate a video frame commodity target set and a commodity image commodity target set; Respectively performing viewpoint recognition, display form recognition, and target quality recognition on the set of commodity targets in the video frame to obtain a viewpoint score, a display form score, and a target quality score; Input the explanation copy, the video frame product target set, the product image product target set and the product titles in the product set into a multimodal feature extraction model for feature extraction processing, obtain the product image product target feature code and the video frame product target feature code, and calculate the product image product target feature code and the video frame product target feature code to generate a product matching score; The viewing angle score, display format score, target quality score and product matching score are added together to obtain a video frame importance score, and key frame screening is performed in combination with the frame extraction frequency to obtain a key frame set; A frame extraction list is generated according to the key frame set.

4. The video key frame extraction method for intelligent slicing of clothing live broadcast video according to claim 3 is characterized in that: Perform perspective recognition on the commodity target set of the video frame to obtain a perspective score, which is specifically: A classification neural network is used for perspective recognition, wherein the perspective categories include front, back, left and right, and the perspective score is the predicted probability of the front category; When it is determined that multiple commodity targets are extracted from a frame of video, the highest score of all target perspectives is taken as the perspective score of the frame. The calculation formula is: Among them, T f are all the targets identified in the f-th frame, p(t,c) is the predicted probability that target t belongs to category c, and S(f) is the view score of the f-th frame.

5. The video key frame extraction method for intelligent slicing of clothing live broadcast video according to claim 3 is characterized in that: The display form of the commodity target set in the video frame is identified to obtain a display form score, which is specifically: A classification neural network is used to identify the display form, wherein the display form categories include try-on and non-try-on, and the display form score is the predicted probability of the try-on category; When it is determined that multiple commodity targets are extracted from a frame of video, the highest score among all target display forms is used as the display form score of the frame.

6. The video key frame extraction method for intelligent slicing of clothing live broadcast video according to claim 3 is characterized in that: The target quality is identified for the set of commodity targets in the video frame to obtain a target quality score, which is specifically: Using a binary classification neural network to identify target quality, wherein the target quality categories include good and bad, and the target quality score is the predicted probability of the good category; When it is determined that multiple product targets are extracted from a frame of video, the lowest quality score of all targets is used as the target quality score of the frame; When it is determined that the predicted probability of any product target quality being a bad category is greater than 0.5, the frame is discarded and does not enter the subsequent screening stage.

7. The video key frame extraction method for intelligent slicing of clothing live broadcast video according to claim 3 is characterized in that: The explanation copy, the video frame product target set, the product image product target set and the product title in the product set are input into the multimodal feature extraction model for feature extraction processing to obtain the product image product target feature code and the video frame product target feature code, and the product image product target feature code and the video frame product target feature code are calculated to generate a product matching score, which is specifically: Use a multimodal feature extraction model to perform feature encoding processing on each product target on the product side and the video frame side to obtain product target feature encoding of the product image and the video frame. The model input on the video frame side is the explanation copy within a 20-second time window 10 seconds before and after the corresponding time of the product target image and the video frame in the current video. The model input on the product side is the product target image on the product side and the title of the current product. According to the commodity target feature coding of the commodity image and the commodity target feature coding of the video frame, the pairwise matching degree of all commodity targets on the video frame side and the commodity side is calculated, and the matching value with the highest matching degree is taken as the commodity matching score of the video frame.

8. The video key frame extraction method for intelligent slicing of clothing live broadcast video according to claim 7 is characterized in that: The product matching score represents the score of the matching degree between the video frame and the list of products to be segmented.

9. The video key frame extraction method for intelligent slicing of clothing live broadcast video according to claim 3 is characterized in that: The multimodal feature extraction model uses an open source multimodal model clip, wherein the matching degree measurement function uses cosine similarity.

10. A video key frame extraction device for intelligent slicing of live video of clothing shows, characterized in that: include: A frame division unit is used to obtain a live video, an explanation copy and a product set, and perform frame processing on the live video to generate a video frame set; The key frame screening unit is used to perform key frame screening processing on the video frame set, explanation copy and product set to generate a frame extraction list, wherein each frame of the video frame set is scored, and key frames are selected according to the score and the frame extraction frequency. The scoring process includes the score scoring process of the video frame itself and the score scoring process of matching with the product.

Citation Information

Cited By

  • Video key frame quality enhancement method and device, equipment and storage medium

    CN121305446A