Commodity video frame matching method and device applied to clothing special field live video intelligent slices
By combining a multimodal large model with perspective, display form similarity and product matching similarity, the background interference problem of matching video frames and product maps in live video intelligent slices is solved, and a more accurate product matching effect is achieved.
Patent Information
- Application Number
- CN202411083262.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, in the live video intelligent slice, the similarity calculation between the video frame and the product picture has great background interference, which is affected by the anchor's posture, and it is difficult to accurately distinguish the current products, resulting in poor matching effects.
Using a multimodal large model combining perspective, display form similarity and product matching similarity, integrating dual parameters, taking into account product target pictures, text image multimodal information of explanation copy and product title, to accurately match video frames with products .
It improves the precise matching effect between video frames and products, reduces background interference, enhances the ability to identify products, and ensures that the products match the explanation video frame to which they belong.
Smart Images

Figure CN119094802A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of commodity video frame matching, and in particular to a commodity video frame matching method and device applied to intelligent slicing of live video of a clothing show. Background Art
[0002] The task of intelligent slicing of live video refers to automatically slicing video clips related to products, such as product introduction, Q&A and display, from the recorded video in the live broadcast room, and giving the products involved in the clip. Video processing is generally converted into an image processing task for video frame processing. First, the video is frame-extracted, and then each product is matched with its corresponding video frame set. Finally, the video frame set matched by each product is merged into several time segments, that is, the explanation clip corresponding to the product is obtained. Among them, matching the video frame set corresponding to the product is the top priority of the entire task, which directly affects the effect of the final slicing. Generally, the similarity between the video frame and the product image is calculated, and then the video frame with higher similarity is used as the video frame set matching the product.
[0003] The current method of directly calculating the similarity between the video frame and the product image has the problem of large background interference and is greatly affected by the host's posture, and sometimes it is impossible to accurately distinguish the product being discussed only by image features.
[0004] In view of this, this application is filed. Summary of the invention
[0005] The present invention provides a commodity video frame matching method and device applied to intelligent slicing of live video of a clothing show, which can at least partially improve the above-mentioned problems.
[0006] To achieve the above object, the present invention adopts the following technical solutions:
[0007] A commodity video frame matching method applied to intelligent slicing of clothing live broadcast video, comprising:
[0008] Acquire a live video and a product set, and perform frame extraction processing on the live video to obtain a video frame set;
[0009] The video frame set and the product set are subjected to similarity calculation processing to generate a product-corresponding explanation video frame set, wherein the similarity between the video frame and the product is calculated, and the explanation video frame corresponding to each product is matched according to the similarity.
[0010] The present invention also provides a commodity video frame matching device for intelligent slicing of live video of a clothing show, which comprises:
[0011] A frame extraction unit, used to obtain a live video and a product set, and perform frame extraction processing on the live video to obtain a video frame set;
[0012] The similarity calculation unit is used to perform similarity calculation processing on the video frame set and the product set to generate a set of explanation video frames corresponding to the products, wherein the similarity between the video frame and the product is calculated, and the corresponding explanation video frame is matched to each product according to the similarity.
[0013] In summary, the product video frame matching method applied to the intelligent slicing of clothing live broadcast videos takes into account the interference factors of product background, the influence of clothing perspective, display form differences, and the semantic information of explanation text and product title in the matching of products and video frames, so as to more accurately match the products to the corresponding explanation video frames. Specifically, the product video frame matching method applied to the intelligent slicing of clothing live broadcast videos combines the perspective, display form similarity and product matching similarity in the calculation of the similarity between products and video frames; in the calculation of product matching scores, the multimodal information of the text images of the product target pictures and the explanation text and product titles is considered; and a multimodal large model is used to expand the text description on the product side, while dual parameter fusion is adopted in the fusion of perspective, display form similarity and product matching similarity. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a flow chart of a commodity video frame matching method for intelligent slicing of live video of a clothing show provided by the first aspect of the present invention;
[0015] Figure 2 It is a flow chart of a commodity video frame matching method applied to intelligent slicing of live video of a clothing show provided by the second aspect of the present invention;
[0016] Figure 3 It is a schematic diagram of a similarity calculation process of a commodity video frame matching method applied to intelligent slicing of a clothing live broadcast video provided by the first embodiment of the present invention;
[0017] Figure 4 It is a module diagram of a commodity video frame matching device for intelligent slicing of live video of a clothing show provided by the second embodiment of the present invention. DETAILED DESCRIPTION
[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0019] refer to Figure 1 , Figure 2As shown, the first embodiment of the present invention discloses a commodity video frame matching method applied to intelligent slicing of clothing live broadcast video, which can be executed by a commodity video frame matching device applied to intelligent slicing of clothing live broadcast video (hereinafter referred to as matching device), and in particular, executed by one or more processors in the matching device to implement the following method:
[0020] S101, obtaining a live video and a product set, and performing frame extraction processing on the live video to obtain a video frame set;
[0021] S102, performing similarity calculation processing on the video frame set and the product set to generate a set of explanation video frames corresponding to the products, wherein the similarity between the video frame and the product is calculated, and the explanation video frame corresponding to each product is matched according to the similarity.
[0022] Specifically, step S102 includes: performing commodity detection processing on the video frame set and the commodity image set in the commodity set to generate a video frame commodity target set and a commodity image commodity target set;
[0023] Respectively performing perspective recognition, display form recognition and target quality recognition on the video frame product target set and the product image product target set to obtain perspective and display form similarity;
[0024] Inputting a set of product images in the product set into a multimodal large model for preprocessing to generate product descriptions;
[0025] Obtaining explanation text and product title, and inputting the explanation text, product image product target set, product description and product title into a multimodal feature extraction model to obtain product image product target feature coding and video frame product target feature coding, performing calculation processing on the product image product target feature coding and video frame product target feature coding to generate product matching similarity;
[0026] The similarity of the viewing angle, display form and product matching is integrated to obtain a similarity score;
[0027] A set of explanation video frames corresponding to the product is generated according to the similarity score.
[0028] In this embodiment, firstly, the live video is extracted to obtain a video frame set, then the similarity between the video frame and the product is calculated, and then the corresponding explanation video frame is matched to each product according to the similarity. Figure 3First, all video frames and product images are detected to obtain the area corresponding to the product, and then the corresponding product target image. It should be noted that multiple products may be detected in the same image, so the same image may correspond to multiple product target images. Then, the product target images on the video side and the product side are respectively identified in terms of perspective, display form and target quality. Low-quality targets are filtered out through the target quality identification results, and the perspective and display form similarity of the product target images on the video side and the product side are calculated. The product matching similarity is to calculate the feature codes on the product side and the video side respectively, and then calculate the similarity between the feature codes. Then, the similarity of perspective, display form and product matching is fused to obtain the similarity score of the final video frame and product.
[0029] Preferably, a binary classification neural network is used for target quality recognition. The target quality types include good and bad, and the target quality score is the predicted probability of the good category. When it is determined that multiple commodity targets are extracted from a frame of video, the lowest score of all target quality scores is used as the target quality score of the frame.
[0030] Preferably, when it is determined that the predicted probability of any bad category in the target quality of the commodity is greater than 0.5, the frame is discarded.
[0031] In this embodiment, the target quality recognition is a binary classification neural network, the categories include good and bad, and the target quality score is the predicted probability of "good". If multiple commodity targets are extracted from a frame of video, the quality score is the lowest score of all target quality scores as the quality score of the frame. It should be noted that if the predicted probability of "bad" in any commodity target quality is greater than 0.5, the frame is discarded.
[0032] Preferably, the video frame commodity target set and the commodity image commodity target set are respectively subjected to perspective recognition and display form recognition to obtain perspective and display form similarity, specifically:
[0033] A viewpoint recognition neural network and a display form recognition neural network are used to perform viewpoint recognition and display form recognition, wherein the viewpoint recognition neural network and the display form recognition neural network are both classification networks, the viewpoint recognition neural network includes four categories: front, back, left and right, and the display form recognition neural network includes two categories: try-on and non-try-on;
[0034] Inputting the product images in the product image product target set into the viewpoint recognition neural network and the display form recognition neural network, and passing through the softmax normalized exponential function to obtain a vector with a length of 4 and a vector with a length of 2 respectively;
[0035] A vector with a length of 4 and a vector with a length of 2 are concatenated to obtain a viewing angle and a display form feature vector, and calculations are performed based on the viewing angle and the display form feature vector to obtain the viewing angle and the display form similarity.
[0036] Preferably, the calculation formula of the viewing angle and display form feature vector X is:
[0037] X = concat(softmax(f view (I)), softmax(f form (I)))
[0038] Among them, f view is the view recognition neural network, f form is the display form recognition neural network, I is the product target graph, and concat is the connection function.
[0039] Preferably, the perspective and display form similarity is the similarity of the perspective and display form feature vectors between the product target image on the product side and the product target image on the video side, and a cosine similarity measurement method is used.
[0040] In this embodiment, the similarity of view angle and display form is obtained through a view angle recognition neural network and a display form recognition neural network. Both networks are classification networks, where the view angle recognition neural network contains four categories: front, back, left and right, and the display form recognition neural network contains two categories: try-on and non-try-on. The product image is input into the two neural networks and after softmax, a vector of length 4 and length 2 are obtained respectively. The two vectors are concatenated to obtain the view angle and display form feature representation. For example, for a product image I, we input it into the view angle recognition neural network and the display form recognition neural network, and represent the view angle recognition neural network as f view , the display form recognition neural network is represented as f form , perspective, the calculation process of the display form feature vector X is as follows:
[0041] X = concat(softmax(f view (I)), softmax(f form (I)))
[0042] The similarity of viewing angle and display form is to calculate the similarity of the viewing angle and display form feature vectors between the product target image on the product side and the product target image on the video side. Any vector similarity measurement method can be used. Here, cosine similarity is taken as an example. The calculation process is as follows:
[0043]
[0044] Preferably, the explanation copy and the product title are obtained, and the explanation copy, the product image product target set, the product description and the product title are input into the multimodal feature extraction model to obtain the product image product target feature code and the video frame product target feature code, and the product image product target feature code and the video frame product target feature code are calculated and processed to generate the product matching similarity, specifically:
[0045] The multimodal feature extraction model is used to perform feature encoding processing on each product target on the product side and the video frame side. The model input on the video frame side is the explanation copy within a 20-second time window 10 seconds before and after the corresponding time of the product target image in the current video and the video frame;
[0046] The product target image is input into the multimodal large model to generate a colloquial product description. The product description is combined with the product title as the text input on the product side, and the product target image is used as the image input and the model input on the product side.
[0047] Preferably, the product matching similarity represents the matching similarity value between the product target image on the video side and the product target image on the product side, the multimodal feature extraction model is an open source multimodal model, and the matching degree measurement function adopts cosine similarity.
[0048] In this embodiment, for the product matching similarity, it represents the matching similarity value between the product target image on the video side and the product target image on the product side. First, each product target on the product side and the video frame side is feature encoded through a multimodal feature extraction model. The model input on the video frame side is the explanation copy in a 20-second time window 10 seconds before and after the corresponding time of the product target image and the video frame of the current video. On the product side, the product target image is first input into a multimodal large model, such as qwen-vl, etc., so that the model generates a colloquial product description, and then the product description is combined with the product title as the product side text input, and the product target image is input as an image input to the multimodal encoding model. The multimodal feature extraction model can use an open source multimodal model, such as clip, etc., or it can be obtained by pre-training. Here, clip is taken as an example, and the matching degree measurement function uses cosine similarity.
[0049] The similarity between the two target product images is obtained by combining the view angle, display form similarity and product matching similarity. Generally speaking, the more similar the view angle and display form are, the higher the similarity between the two images is. For images with different view angles and display forms, even if they are the same product, the similarity may be low. Therefore, the final similarity is negatively correlated with the view angle and display form similarity, and positively correlated with the product matching similarity. vf Similarity S with product matching gm, first perform a nonlinear transformation to map it between 0 and 1, the formula is:
[0050]
[0051] Then calculate the final similarity:
[0052] S total =w1·(1-S′ vf )+w2·S′ gm
[0053] Among them, w1 and w2 are two hyperparameters used to balance the contribution of the two similarities. For multiple product target images in a frame and multiple pictures of a product, the multiple product target images of each picture use the maximum value of the pairwise similarities of the product target images on the video side and the product side as the similarity value of the frame and the product. The calculation process is as follows: Assume that there is a frame F containing n product target images, represented as {F1, F2, ..., Fn}; there is a product G containing m pictures, represented as {G1, G2, ..., Gm}. Each Gj (that is, the jth picture of product G) contains pj product target images, represented as {Gj1, Gj2, ..., Gjpj}. Define S(Fi, Gjk) as the similarity between the i-th product target image in a frame F and the k-th product target image in the j-th picture of product G. Then, the similarity value S(F, G) between the frame F and the product G can be defined as:
[0054]
[0055] A similarity threshold r may be set, and for a product G, all frames with similarities greater than the threshold are taken as a set of video frames recalled for the product G.
[0056] In this embodiment, the product video frame matching method applied to the intelligent slicing of the clothing special live broadcast video combines the view angle, display form similarity and product matching similarity in the calculation of the similarity between the product and the video frame, and fully considers the influence of the view angle and display form factors; in the calculation of the product matching score, the multimodal information of the text image of the product target picture and the explanation copy and the product title is considered, compared with using only image features, the similarity between the explanation product and the target product can be more accurately modeled; at the same time, the multimodal large model is used to expand the product side text description, and the oral explanation copy and the product side text features are aligned in combination with the large model capability; and the dual parameter fusion is adopted in the fusion of the view angle, display form similarity and product matching similarity, so that the final similarity is negatively correlated with the view angle and display form, and positively correlated with the product matching similarity, further alleviating the influence of the view angle and display form factors.
[0057] See also Figure 4The second embodiment of the present invention further provides a commodity video frame matching device for intelligent slicing of clothing live broadcast video, which includes:
[0058] The frame extraction unit 201 is used to obtain a live video and a product set, and perform frame extraction processing on the live video to obtain a video frame set;
[0059] The similarity calculation unit 202 is used to perform similarity calculation processing on the video frame set and the product set to generate a product corresponding explanation video frame set, wherein the similarity between the video frame and the product is calculated, and the corresponding explanation video frame is matched to each product according to the similarity.
[0060] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A commodity video frame matching method for intelligent slicing of live video of clothing special events, It is characterized in that include: Acquire a live video and a product set, and perform frame extraction processing on the live video to obtain a video frame set; The video frame set and the product set are subjected to similarity calculation processing to generate a product-corresponding explanation video frame set, wherein the similarity between the video frame and the product is calculated, and the explanation video frame corresponding to each product is matched according to the similarity.
2. According to claim 1, the commodity video frame matching method for intelligent slicing of clothing live broadcast video, It is characterized in that The video frame set and the product set are similarly calculated to generate a product explanation video frame set, specifically: Performing commodity detection processing on the set of commodity images in the set of video frames and the set of commodities to generate a set of commodity targets in the video frames and a set of commodity targets in the commodity images; Respectively performing perspective recognition, display form recognition and target quality recognition on the video frame product target set and the product image product target set to obtain perspective and display form similarity; Inputting a set of product images in the product set into a multimodal large model for preprocessing to generate product descriptions; Obtaining explanation text and product title, and inputting the explanation text, product image product target set, product description and product title into a multimodal feature extraction model to obtain product image product target feature coding and video frame product target feature coding, performing calculation processing on the product image product target feature coding and video frame product target feature coding to generate product matching similarity; The similarity of the viewing angle, display form and product matching is integrated to obtain a similarity score; A set of explanation video frames corresponding to the product is generated according to the similarity score.
3. According to claim 2, the commodity video frame matching method for intelligent slicing of clothing live broadcast video, It is characterized in that A binary classification neural network is used to identify target quality. The target quality types include good and bad, and the target quality score is the predicted probability of the good category. When it is determined that multiple commodity targets are extracted from a frame of video, the lowest score of all target quality scores is used as the target quality score of the frame.
4. According to claim 3, the commodity video frame matching method for intelligent slicing of clothing live broadcast video, It is characterized in that When it is determined that the predicted probability of any bad category in the product target quality is greater than 0.5, the frame is discarded.
5. According to claim 4, the commodity video frame matching method for intelligent slicing of clothing live broadcast video, It is characterized in that The video frame commodity target set and the commodity image commodity target set are respectively subjected to perspective recognition and display form recognition to obtain perspective and display form similarity, specifically: A viewpoint recognition neural network and a display form recognition neural network are used to perform viewpoint recognition and display form recognition, wherein the viewpoint recognition neural network and the display form recognition neural network are both classification networks, the viewpoint recognition neural network includes four categories: front, back, left and right, and the display form recognition neural network includes two categories: try-on and non-try-on; Inputting the product images in the product image product target set into the viewpoint recognition neural network and the display form recognition neural network, and passing through the softmax normalized exponential function to obtain a vector with a length of 4 and a vector with a length of 2 respectively; A vector with a length of 4 and a vector with a length of 2 are concatenated to obtain a viewing angle and a display form feature vector, and calculations are performed based on the viewing angle and the display form feature vector to obtain the viewing angle and the display form similarity.
6. The commodity video frame matching method for intelligent slicing of clothing live broadcast video according to claim 5, It is characterized in that The calculation formula of the viewing angle and display form feature vector X is: X=concat(softmax(f view (I)),softmax(f form (I))) Among them, f view is the view recognition neural network, f form is the display form recognition neural network, I is the product target graph, and concat is the connection function.
7. The commodity video frame matching method for intelligent slicing of clothing live broadcast video according to claim 2, It is characterized in that The perspective and display form similarity is the similarity of the perspective and display form feature vectors between the product target image on the product side and the product target image on the video side, and is measured using a cosine similarity method.
8. The commodity video frame matching method for intelligent slicing of clothing live broadcast video according to claim 2, It is characterized in that Obtain explanation text and product title, and input the explanation text, product image product target set, product description and product title into the multimodal feature extraction model to obtain product image product target feature coding and video frame product target feature coding, calculate and process the product image product target feature coding and video frame product target feature coding to generate product matching similarity, specifically: The multimodal feature extraction model is used to perform feature encoding processing on each product target on the product side and the video frame side. The model input on the video frame side is the explanation copy within a 20-second time window 10 seconds before and after the corresponding time of the product target image in the current video and the video frame; The product target image is input into the multimodal large model to generate a colloquial product description. The product description is combined with the product title as the text input on the product side, and the product target image is used as the image input and the model input on the product side.
9. The commodity video frame matching method for intelligent slicing of clothing live broadcast video according to claim 2, It is characterized in that The product matching similarity represents the matching similarity value between the product target image on the video side and the product target image on the product side. The multimodal feature extraction model is an open source multimodal model, and the matching degree measurement function adopts cosine similarity.
10. A commodity video frame matching device for intelligent slicing of live video of clothing special events, It is characterized in that include: A frame extraction unit, used to obtain a live video and a product set, and perform frame extraction processing on the live video to obtain a video frame set; The similarity calculation unit is used to perform similarity calculation processing on the video frame set and the product set to generate a set of explanation video frames corresponding to the products, wherein the similarity between the video frame and the product is calculated, and the corresponding explanation video frame is matched to each product according to the similarity.
