Method for key content segmentation of a video and electronic device
By using an end-to-end multimodal deep learning model to process videos and images, this technology addresses the shortcomings in intelligence and versatility of existing technologies for segmenting key video content, and enables effective segmentation and tracking of multiple objects and complex scenes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TAOBAO CHINA SOFTWARE
- Filing Date
- 2023-02-24
- Publication Date
- 2026-07-07
AI Technical Summary
Existing technologies rely on predefined object categories or annotations in the first frame for video key content segmentation, resulting in insufficient intelligence and versatility, and difficulty in handling scenarios with multiple objects or no annotations.
An end-to-end multimodal deep learning model based on the Transformer attention mechanism is adopted. By receiving target video and target image as input, feature information is extracted using a dual encoder, fused and decoded to identify and track key content, and output segmentation results.
It achieves key content segmentation without the need for predefined object categories and first frame annotations, improving the intelligence and versatility of video processing and enabling it to handle multiple objects and complex scenes.
Smart Images
Figure CN116342880B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video processing technology, and in particular to methods and electronic devices for segmenting key content in videos. Background Technology
[0002] With the rise of short videos and live streaming, video content analysis has become an increasingly important research direction. A crucial aspect of video content analysis is segmenting key elements such as people and objects to facilitate editing and trimming. For example, in videos related to product demonstrations, post-production may require highlighting product images as key elements (e.g., magnifying parts of the product image, adding flash effects, adding text, etc.), or identifying key elements like product images, performing background removal, and then replacing the background, etc.
[0003] One existing approach to segmenting key content in video involves simultaneously tracking and segmenting multiple object categories. This means pre-defining various identifiable object categories and then tracking and segmenting objects of any category. However, this method relies on the pre-defined object categories and cannot track or segment objects not pre-defined in the system. Another existing approach involves manually annotating the key content in the first frame of the video, and then using an algorithm to segment the key content in subsequent frames based on the annotation information from the first frame. This method overcomes the limitation of object categories; however, it relies on manual annotation of the first frame. If the first frame does not contain the content of the object of interest, it is difficult to implement, and it is also difficult to simultaneously process multiple objects.
[0004] Therefore, how to improve the intelligence and versatility of video key content segmentation processing has become a technical problem that needs to be solved by those skilled in the art. Summary of the Invention
[0005] This application provides a method and electronic device for segmenting key content in videos, which can improve the intelligence and versatility of key content segmentation processing in videos.
[0006] This application provides the following solution:
[0007] A method for segmenting key content in a video includes:
[0008] Receive the target video to be segmented for key content, and the target image containing the key content of interest;
[0009] Identify target key content that matches key content in the target image from the target video;
[0010] The target key content is segmented and tracked, and the key content segmentation results are output.
[0011] The step of determining the target key content from the target video that matches the key content in the target image includes:
[0012] From the target video, obtain feature information of at least one candidate key content related to key content in the target image;
[0013] By matching the feature information of the at least one candidate key content with the image features of key content in the target image, the target key content that meets the confidence criteria is determined from the at least one candidate key content;
[0014] The segmentation and tracking of the target key content, and the output of the key content segmentation results, include:
[0015] The target key content is segmented and tracked based on its corresponding feature information, and the key content segmentation result is output.
[0016] The feature information of the candidate key content includes: whether the candidate key content appears in multiple image frames of the target video, and the location information of the region where it appears.
[0017] Wherein, obtaining feature information of at least one candidate key content related to key content in the target image from the target video includes:
[0018] Feature extraction is performed on the target image and the target video respectively, and the extracted image feature information and video feature information are fused to obtain fused feature information, so that the parts of the target video related to the key content are activated and the irrelevant parts are ignored;
[0019] By decoding the fused feature information, feature information of at least one candidate key content identified from the target video is determined.
[0020] The step of determining the feature information of at least one candidate key content identified from the target video by decoding the fused feature information includes:
[0021] The fused feature information is decoded using a deep learning model based on an attention mechanism in order to determine the feature information of at least one candidate key content identified from the target video;
[0022] The target number of random vectors are used as the first input information of the deep learning model, and the fused feature information is used as the second and third input information of the deep learning model.
[0023] The step of segmenting and tracking based on the feature information corresponding to the target key content and outputting the key content segmentation result includes:
[0024] Based on the feature information corresponding to the target key content, a binary mask of the target key content in multiple image frames of the target video is obtained; wherein, the binary mask is used to represent the part of the image frame that belongs to the target key content with a first value, and the part that does not belong to the target key content with a second value;
[0025] Based on the binarized mask, the segmentation results of the target key content in multiple image frames of the target video are output.
[0026] The step of outputting the segmentation result of the target key content in multiple image frames of the target video based on the binarized mask includes:
[0027] Based on the binarized mask, the target key content is instantiated and tracked across multiple image frames to obtain the binarized mask corresponding to each instance of the target key content.
[0028] Based on the binary masks corresponding to the multiple instances, the segmentation results of the multiple instances of the target key content in the multiple image frames of the target video are output.
[0029] The target video includes videos related to product explanations, and the target image includes images containing product information that needs to be of interest.
[0030] The target video includes videos related to the person, and the target image includes images containing information about the person of interest.
[0031] A method for generating product demonstration videos includes:
[0032] Acquire the target video for product content segmentation, and the target image containing the product information of interest.
[0033] Identify target products that match the products in the target image from the target video;
[0034] The target product is segmented and tracked, and the product content segmentation result is output.
[0035] The target video is edited based on the product content segmentation results to generate an explanatory video about the target product.
[0036] An apparatus for segmenting key content in a video, comprising:
[0037] The input information receiving unit is used to receive the target video to be segmented for key content, and the target image containing the key content of interest.
[0038] The target key content determination unit is used to determine the target key content that matches the key content in the target image from the target video;
[0039] The segmentation result output unit is used to segment and track the target key content and output the key content segmentation result.
[0040] An apparatus for generating product demonstration videos, comprising:
[0041] The input information receiving unit is used to acquire the target video to be segmented into product content, and the target image containing the product information of interest.
[0042] A target product determination unit is used to determine a target product that matches the product in the target image from the target video;
[0043] The segmentation result output unit is used to segment and track the target product and output the product content segmentation result;
[0044] The editing and processing unit is used to edit the target video according to the product content segmentation results to generate an explanatory video about the target product.
[0045] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.
[0046] An electronic device, comprising:
[0047] One or more processors; and
[0048] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the preceding descriptions.
[0049] According to the specific embodiments provided in this application, the following technical effects are disclosed:
[0050] According to the embodiments of this application, when it is necessary to segment key content in a target video, the target video and a target image containing the key content of interest can be used as input information. Then, target key content related to the key content in the target image can be determined from the target video, segmented and tracked, and the key content segmentation result can be output. This method no longer relies on predefined object categories or annotation operations on the first frame, thus making the solution more intelligent and more versatile.
[0051] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0053] Figure 1 This is a schematic diagram of the system architecture provided in the embodiments of this application;
[0054] Figure 2 This is a flowchart of the first method provided in the embodiments of this application;
[0055] Figure 3 This is a flowchart of the second method provided in the embodiments of this application;
[0056] Figure 4 This is a schematic diagram of the first device provided in the embodiments of this application;
[0057] Figure 5 This is a schematic diagram of the second device provided in the embodiments of this application;
[0058] Figure 6 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation
[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0060] To facilitate understanding of the technical solutions provided in this application, it should first be noted that in the implementation scheme for object tracking and segmentation based on the annotation results of the first frame of a video described in the background section, if the first frame of a specific video does not include key content such as the product of interest, it is difficult to achieve segmentation and tracking of key content using this method. Furthermore, if the first frame contains the content of the object of interest, and there are no other objects besides that object in the video content, this scheme can usually achieve good object segmentation results. The relevant algorithm model only needs to perform binarization processing on the parts related to the annotated content in subsequent frames based on the annotation results in the first frame. However, in practical applications, there are often complex videos that may include multiple objects. For example, in videos recorded during product live streaming, multiple products may simultaneously enter the frame while the host is introducing the product, including multiple identical or similar products, etc. In this case, the same video may contain the content of multiple products. Therefore, the above-mentioned method of segmenting key content by annotating the first frame usually fails to output accurate segmentation results.
[0061] To address the above issues, this application provides a corresponding solution. This solution improves existing algorithm models, allowing them to receive two inputs simultaneously: a target video for which key content segmentation is needed, and an image containing information about the key content. For example, assuming the target video is related to a product introduction and the goal is to segment content belonging to a specific product, the target video can be input into the algorithm model. Simultaneously, a target image, such as a photo of the product, can also be input. Thus, when performing key content segmentation on a video, only an image related to the specific key content needs to be prepared and input along with the video into the model for end-to-end computation. This outputs the key content segmentation result, eliminating reliance on the annotation of the first frame of the video and limitations on the category of key content, enabling the tracking and segmentation of any key content.
[0062] From a system architecture perspective, see Figure 1This application provides a system or tool for video key content segmentation. This system or tool can exist independently or be integrated into a related video processing system or tool (e.g., a tool for post-processing video). Using this key content segmentation system or tool, end-to-end key content segmentation can be completed using the target video and target image as input. Since the target image can express the information of the specific key content to be segmented, without relying on predefined object categories or annotation operations on the first frame, the solution is more intelligent and versatile. In specific implementation, the structure of the algorithm model can be determined and trained using a labeled training dataset. Then, the trained model can be used to complete the aforementioned end-to-end key content segmentation process.
[0063] In one specific implementation, such as Figure 1 As shown, the specific algorithm model can be an end-to-end multimodal deep learning model based on attention mechanisms such as Transformer. The input of this model can employ a dual-encoder structure (a first encoding module and a second encoding module) to support two inputs: a target video and a target image. Then, the model can extract feature information of at least one candidate key content related to key content in the target image from the target video. Specifically, video features and image features can be fused using structures such as Transformer, and then the Transformer decoding module can determine the feature information of one or more candidate key contents from the fused features. Afterwards, an object matching module can select target key contents that meet the matching criteria with key content in the target image. Finally, based on the feature information of the target key contents, the key content segmentation result can be obtained. In cases where the video includes multiple instances of objects, the output segmentation result can include a multi-layer segmentation network. The first segmentation network can be used to obtain the segmentation result for each image frame of the video, the second segmentation network can be used for further instantiation tracking processing, and the third segmentation network can be used to output one or more instantiation masks of the key content.
[0064] The specific implementation schemes provided in the embodiments of this application will be described in detail below.
[0065] Example 1
[0066] First, this embodiment provides a method for segmenting key content in a video from the perspective of the aforementioned video key content segmentation system or tool. See [link to documentation]. Figure 2 The method may include:
[0067] S201: Receive the target video to be segmented for key content, and the target image containing the key content of interest.
[0068] In this embodiment, the input information includes two types: one is the target video for which key content segmentation is required, and the other is the target image containing the key content of interest. For example, users with video processing needs can prepare the target video and target image in advance so that both can be input into the model for processing.
[0069] The key content can specifically include objects, people, etc., and can be related to the actual application scenario. For example, suppose a video is about explaining a product, and we need to segment out the content related to a specific product for detailed magnification; then the product would be the key content to be segmented. Or, suppose a video is a clip from a movie or TV show about a person, and we need to segment out the content related to that person and change the video background; then that person would be the key content to be segmented, and so on.
[0070] The target image can be a photograph obtained beforehand by taking pictures of the actual object corresponding to the key content. In practice, to achieve the ideal segmentation effect, the quality of the target image should be ensured as much as possible. For example, the key content of interest can occupy the main part of the target image, and the background can be kept as simple and clean as possible to reduce the impact of noise in the target image on the segmentation result.
[0071] From the system side, two interfaces can be provided to users for inputting videos and images. Alternatively, an input interface can be provided to users in a productized way. The interface can include two input controls for inputting the target video and the target image, respectively. In this way, users can use this input interface to upload the target video and the target image, and so on.
[0072] For existing image processing or video processing models, since they typically only support one type of input information, they cannot be directly applied. Therefore, in this application embodiment, the input end of the specific model can be improved to have a dual-encoder structure for simultaneously processing the target video and the target image. That is, the model can include a first encoder and a second encoder. After receiving the target video and the target image, the target video can be input into the first encoder, the target image into the second encoder, and so on.
[0073] S202: Determine the target key content from the target video that is related to the key content in the target image.
[0074] After receiving the target video and target image, the target key content related to the key content in the target image can be determined from the target video. That is, the region containing the target key content that matches the key content shown in the target image is determined in multiple image frames of the target video, so as to perform subsequent segmentation and tracking processing.
[0075] In one specific implementation, to determine the aforementioned target key content, feature information of at least one candidate key content related to the key content in the target image can first be obtained from the target video. Then, by matching the feature information of the at least one candidate key content with the image features of the key content in the target image, the target key content with the required confidence level is determined from the at least one candidate key content. That is, assuming the target image contains key content related to an object, and the target video may contain multiple objects, some of which may be related to the object in the target image, and some may be unrelated. Among the related objects, some may indeed be the object to be segmented, while others may only be similar, and so on. Therefore, in this step, features of one or more objects that may be related to the target object can be identified first from the target video. Through this step, objects in the target video that are clearly unrelated to the object of interest can be ignored.
[0076] In specific implementation, the feature information of candidate key content can include whether the candidate key content appears in multiple image frames of the target video, and the location information of the region where it appears. That is, the feature information of candidate key content can be expressed by an N-dimensional vector, where N is the number of frames in the target video. For example, assuming the target video has 100 frames, each candidate key content can be represented by a 100-dimensional vector. Each element in this 100-dimensional vector can still be a specific feature vector, used to express whether the current candidate key content appears in the current frame, and if so, the location of the current key content within that image frame, etc.
[0077] Specifically, there are several ways to obtain the feature information of the aforementioned candidate key content. For example, one approach is to first extract features from both the target image and the target video, and then fuse the extracted image feature information with the video feature information to obtain fused feature information. This activates the parts of the target video related to the key content while ignoring irrelevant parts. Subsequently, the fused feature information can be decoded to determine the feature information of at least one candidate key content identified from the target video.
[0078] Specifically, feature extraction from the target video and target image can be achieved using the aforementioned dual-encoder. The first encoder, upon receiving the target video, processes it into three-dimensional features, including length, width, and time dimensions. The second encoder, upon receiving the target image, processes it into two-dimensional features, including length and width dimensions. The two encoders can process in parallel, forming a dual-encoder structure. After feature extraction from the target video and target image, the two sets of features can be fused. Specifically, to fuse video and image features, in one implementation, attention-based model structures such as Transformer can be used to combine the two sets of features. It should be noted that since the feature dimensions output by the first and second encoders are different, during the fusion process, the image features can be processed to have the same feature dimensions as the video features. In one approach, this process can be achieved by expanding the image features. Specifically, video features, compared to image features, have an additional temporal dimension. This means that feature extraction needs to be performed on multiple image frames within the video. The image features of these multiple frames constitute the video's features. Thus, if a video consists of N (N is a natural number greater than 1) image frames, the first encoder can output an N-dimensional feature vector. Each element of this feature vector is still a specific feature vector describing the image features (including length and width dimensions) of a specific image frame. Therefore, to enable the fusion of video and image features, the image features can be expanded by copying them N times. This ensures that the video and image features have the same feature dimensions, allowing for subsequent fusion.
[0079] Through the aforementioned fusion process, the parts of the target video that are relevant to the key content in the target image can be activated, while irrelevant parts are ignored. It should be noted that this capability can be acquired by pre-training the model. That is, in practical applications, before performing key content segmentation as described in this embodiment, a model structure can be pre-established and trained using a training dataset. This training dataset can include multiple video-image pairs, and the videos can also contain annotations indicating the parts of the video frames that are relevant to key content such as objects in the images. Supervised learning of the model using the training data enables the algorithm to acquire the aforementioned ability to extract and fuse video and image features. Of course, in practice, the algorithm may involve decoding the fused features, matching candidate key content, and outputting segmentation results, in addition to the feature extraction and fusion parts. During model training, these multiple parts can be jointly trained to train multiple components of the model using the same training dataset.
[0080] After fusing video and image features, a fused feature vector is obtained. Since this feature vector can only express which parts of multiple image frames in the video might be relevant to the key content of interest and which are irrelevant, it belongs to a relatively low-order feature and cannot be used directly. Therefore, higher-order features containing key concepts such as objects can be obtained by decoding the fused features. This allows the feature vector to express which parts of the video represent an object, how many objects there are, and approximately which parts of each frame contain objects, etc.
[0081] To achieve the above objectives, a decoder using attention mechanisms such as Transformer can also be used. Specifically, since such decoders typically require multiple inputs, various information components can be constructed to meet the decoder's input conditions. For example, a Transformer decoder typically requires three input components: Query, Key, and Value. In constructing this information, a target number of random vectors can be used as the first input to the deep learning model, and the fused feature information can be used as the second and third inputs.
[0082] The target number can be expressed as a pre-defined fixed value, representing the maximum number of key elements that may exist in the video. For example, assuming that the video contains no more than 10 objects, the target number can be set to 10. Then, 10 vectors can be randomly generated as the query for the Transformer decoder. The fused features obtained earlier can serve as both the key and value for the decoder. The Transformer decoder can then decode the object-level feature sequences, which serve as candidate key elements identified by the model from the input target video. In other words, the decoder output can be multiple feature information, each corresponding to a candidate key element. The number of features output by the decoder is related to the number of the first input information. For example, assuming that the video is expected to contain a maximum of 10 objects, 10 random vectors can be constructed as the first input information for the decoder, and the decoder can ultimately output 10 feature vectors. Of course, not all videos may contain 10 objects; therefore, some of the 10 feature vectors output by the decoder may be empty. For example, if there are only 5 objects in a video that are related to the object that needs to be focused on, then of the 10 feature vectors output by the decoder, only 5 vectors may have values, while the other feature vectors may be empty, such as having 0 values in all dimensions, etc.
[0083] After the above steps, feature information of one or more candidate key contents is obtained. That is, candidate contents in the video that may be related to the key contents that need to be focused on are obtained. However, some of these candidate contents may only be close to the actual key contents that need to be focused on, and may not be the real key contents that need to be focused on. Therefore, the target key contents that meet the confidence conditions can be determined from the at least one candidate key contents by matching the feature information of the at least one candidate key contents with the image features of the key contents in the target image.
[0084] In other words, assuming we obtain feature vectors for five candidate key elements, only one of them may be the truly relevant target key element. The purpose of this step is to identify this truly relevant target key element. Specifically, this can be achieved by matching the feature vectors of each of the five candidate key elements with the image features of the target key element in the target image. The matching score represents the confidence level that these five candidate key elements belong to the truly relevant key element. Those with higher confidence scores or scores exceeding a certain threshold can be identified as the target key element. The image features of the key element in the target image can be extracted by a specific encoder in step S202.
[0085] S203: Segment and track the target key content, and output the key content segmentation result.
[0086] After identifying the target key content, since the specific target key content is still expressed by feature vectors, the key content segmentation result can be output based on the feature information corresponding to the target key content. This segmentation result is more visual and more easily seen and understood by the human eye. For example, the location of the specific key content in multiple image frames of the target video can be marked, and so on.
[0087] In practice, there are multiple ways to achieve this segmentation result output based on the feature information corresponding to the target key content. For example, in one specific implementation, the binarized mask of the target key content in multiple image frames of the target video can be obtained first based on the feature information corresponding to the target key content. The part of the image frame that belongs to the target key content can be represented by a first value (e.g., 1), and the part that does not belong to the target key content can be represented by a second value (e.g., 0). Then, the segmentation result of the target key content in multiple image frames of the target video can be output based on the binarized mask.
[0088] It's important to note that in practice, a single video may contain multiple instances of the same key content. For example, suppose the target video is recorded during a live stream, and the focus is on a cup. The input target image includes an image of the cup, but two cups might actually appear in the video. For instance, one cup (model 1) might be on the table, and the streamer might be holding another cup (model 2). These two cups are two instances of the same cup. Furthermore, after obtaining the binarized masks of the key content across multiple image frames of the target video, these masks are scattered across multiple image frames, not connected sequentially, and do not correspond to specific instances. For example, suppose two cups are segmented in one image frame, and two cups are also segmented in the next image frame. However, how the two cups in these two image frames correspond cannot be directly determined using the aforementioned binarized masks.
[0089] Therefore, in this embodiment of the application, in addition to segmenting the specific target key content, different instances can also be distinguished. Specifically, the target key content can be instantiated and tracked across multiple image frames based on the binarized mask to obtain binarized masks corresponding to multiple instances of the target key content. Then, based on the binarized masks corresponding to the multiple instances, the segmentation results of the multiple instances of the target key content in multiple image frames of the target video can be output, ultimately outputting the segmentation results of one or more instances of the target key content.
[0090] The aforementioned processes of generating the binarized mask, instantiation tracking, and outputting the final segmentation result can all be accomplished using a convolutional neural network. Specifically, in one implementation, after determining the feature information corresponding to the target key content, the first layer of the segmentation network is input to obtain the segmentation result for each frame of the video. Then, the second layer of the segmentation network performs further instantiation tracking based on the segmentation result. Finally, the third layer of the segmentation network outputs the binary segmentation mask for each relevant key content in each frame of the video, ultimately outputting one or more instantiation masks for the key content.
[0091] The above provides a detailed description of the method for segmenting key content in videos according to the embodiments of this application. In practical applications, this method can have various specific application scenarios. For example, in one scenario, the specific target video can be a video related to product explanation. In this case, the specific target image can contain the product information that needs attention, such as a photo of the product. Additionally, as mentioned above, the specific target video can also be a video related to a person, and the specific target image can be a photo of that person, and so on.
[0092] In summary, through the embodiments of this application, when it is necessary to segment key content in a target video, the target video and a target image containing the key content of interest can be used as input information. Then, target key content related to the key content in the target image can be determined from the target video, segmented and tracked, and the key content segmentation result can be output. This method no longer relies on predefined object categories or annotation operations on the first frame; therefore, the solution is more intelligent and more versatile.
[0093] Example 2
[0094] This second embodiment applies the video key content segmentation method described in the first embodiment to the product explanation video processing process, providing a method for generating product explanation videos. See [link to documentation]. Figure 3The method may include:
[0095] S301: Obtain the target video for product content segmentation and the target image containing the product information of interest;
[0096] The target video may include a video recorded and captured during a product livestream. Of course, in practical applications, it can also be a video obtained through other means, such as a specially recorded explanatory video by the merchant, which can then be post-processed using the solution provided in this application embodiment, and so on.
[0097] S302: Determine the target product that matches the product in the target image from the target video;
[0098] S303: Perform segmentation and tracking processing on the target product, and output the product content segmentation result;
[0099] S304: Edit the target video based on the product content segmentation results to generate an explanatory video about the target product.
[0100] For the parts of this embodiment that are not described in detail, please refer to the description in the foregoing embodiment one and other parts of this specification, which will not be repeated here.
[0101] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).
[0102] Corresponding to Embodiment 1, this application also provides an apparatus for segmenting key content in a video, see [link to embodiment]. Figure 4 The device may include:
[0103] The input information receiving unit 401 is used to receive the target video to be segmented for key content, and the target image containing the key content of interest.
[0104] The target key content determination unit 402 is used to determine target key content that matches the key content in the target image from the target video;
[0105] The segmentation result output unit 403 is used to segment and track the target key content and output the key content segmentation result.
[0106] In specific implementation, the target key content determination unit may include:
[0107] The candidate key content feature extraction subunit is used to obtain feature information of at least one candidate key content related to key content in the target image from the target video;
[0108] A matching subunit is used to determine the target key content with a confidence level that meets the conditions from the at least one candidate key content by matching the feature information of the at least one candidate key content with the image features of the key content in the target image;
[0109] At this point, the segmentation result output unit can be specifically used for:
[0110] The target key content is segmented and tracked based on its corresponding feature information, and the key content segmentation result is output.
[0111] The feature information of the candidate key content includes: whether the candidate key content appears in multiple image frames of the target video, and the location information of the region where it appears.
[0112] In specific implementation, the candidate key content feature extraction subunit may include:
[0113] The feature fusion subunit is used to extract features from the target image and the target video respectively, and fuse the extracted image feature information with the video feature information to obtain fused feature information, so as to activate the parts of the target video related to the key content and ignore the irrelevant parts;
[0114] The decoding subunit is used to determine the feature information of at least one candidate key content identified from the target video by decoding the fused feature information.
[0115] In one implementation, the decoding subunit can specifically be used for:
[0116] The fused feature information is decoded using a deep learning model based on an attention mechanism in order to determine the feature information of at least one candidate key content identified from the target video;
[0117] The target number of random vectors are used as the first input information of the deep learning model, and the fused feature information is used as the second and third input information of the deep learning model.
[0118] Specifically, the segmentation result output unit can be used for:
[0119] Based on the feature information corresponding to the target key content, a binary mask of the target key content in multiple image frames of the target video is obtained; wherein, the binary mask is used to represent the part of the image frame that belongs to the target key content with a first value, and the part that does not belong to the target key content with a second value;
[0120] Based on the binarized mask, the segmentation results of the target key content in multiple image frames of the target video are output.
[0121] In addition, the segmentation result output unit can also be used for:
[0122] Based on the binarized mask, the target key content is instantiated and tracked across multiple image frames to obtain the binarized mask corresponding to each instance of the target key content.
[0123] Based on the binary masks corresponding to the multiple instances, the segmentation results of the multiple instances of the target key content in the multiple image frames of the target video are output.
[0124] In one application scenario, the target video includes a video related to product explanation, and the target image includes an image containing product information that needs to be of interest.
[0125] Alternatively, in another application scenario, the target video includes: a video related to a person, and the target image includes: an image containing information about the person of interest.
[0126] Corresponding to Embodiment 2, this application also provides an apparatus for generating product explanation videos, see [link to embodiment]. Figure 5 The device may include:
[0127] The input information receiving unit 501 is used to acquire the target video to be segmented into product content, and the target image containing the product information of interest.
[0128] The target product determination unit 502 is used to determine a target product that matches the product in the target image from the target video;
[0129] The segmentation result output unit 503 is used to segment and track the target product and output the product content segmentation result;
[0130] The editing processing unit 504 is used to edit the target video according to the product content segmentation result to generate an explanatory video about the target product.
[0131] The target video includes: a video recorded and captured during a product live stream.
[0132] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.
[0133] And an electronic device, comprising:
[0134] One or more processors; and
[0135] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.
[0136] in, Figure 6 An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.
[0137] The processor 610 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in this application.
[0138] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, and a critical content segmentation processing system 625, etc. The aforementioned critical content segmentation processing system 625 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 620 and executed by the processor 610.
[0139] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.
[0140] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0141] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.
[0142] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.
[0143] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0144] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0145] The method and electronic device for segmenting key content in video provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for segmenting key content in a video, characterized in that, include: Receive the target video to be segmented for key content, and the target image containing the key content of interest; Identify target key content that matches key content in the target image from the target video; This includes: extracting features from the target video and the target image using a dual-channel encoder, so that the target video is processed into three-dimensional features including length, width, and time, and the target image is processed into two-dimensional features including length and width. Based on the number of image frames N in the target video, the two-dimensional features corresponding to the target image are copied into N copies so that the video features and image features have the same feature dimension before fusion processing is performed. This process activates the parts of the target video that are related to the key content that needs attention in the target image, while ignoring irrelevant parts. Here, N is a natural number greater than 1. The fused feature information is decoded using an attention-based deep learning model to determine the feature information of at least one candidate key content identified from the target video; wherein, a random vector of the target number is used as the first input information of the deep learning model, and the feature information obtained after the fusion process is used as the second and third input information of the deep learning model. By matching the feature information of the at least one candidate key content with the image features of key content in the target image, the target key content that meets the confidence criteria is determined from the at least one candidate key content; The target key content is segmented and tracked, and the key content segmentation results are output.
2. The method according to claim 1, characterized in that, The segmentation and tracking of the target key content, and the output of the key content segmentation results, include: The target key content is segmented and tracked based on its corresponding feature information, and the key content segmentation result is output.
3. The method according to claim 2, characterized in that, The feature information of the candidate key content includes: whether the candidate key content appears in multiple image frames of the target video, and the location information of the region where it appears.
4. The method according to claim 2, characterized in that, The step of segmenting and tracking based on the feature information corresponding to the target key content and outputting the key content segmentation result includes: Based on the feature information corresponding to the target key content, a binary mask of the target key content in multiple image frames of the target video is obtained; wherein, the binary mask is used to represent the part of the image frame that belongs to the target key content with a first value, and the part that does not belong to the target key content with a second value; Based on the binarized mask, the segmentation results of the target key content in multiple image frames of the target video are output.
5. The method according to claim 4, characterized in that, The step of outputting the segmentation results of the target key content in multiple image frames of the target video based on the binarized mask includes: Based on the binarized mask, the target key content is instantiated and tracked across multiple image frames to obtain the binarized mask corresponding to each instance of the target key content. Based on the binary masks corresponding to the multiple instances, the segmentation results of the multiple instances of the target key content in the multiple image frames of the target video are output.
6. The method according to any one of claims 1 to 5, characterized in that, The target video includes: a video related to product explanation, and the target image includes: an image containing product information that needs to be of interest.
7. The method according to any one of claims 1 to 5, characterized in that, The target video includes: videos related to the person, and the target image includes: images containing information about the person to be of interest.
8. A method for generating product demonstration videos, characterized in that, include: Acquire the target video for product content segmentation, and the target image containing the product information of interest. Identify target products that match the products in the target image from the target video; This includes: extracting features from the target video and the target image using a dual-channel encoder, so that the target video is processed into three-dimensional features including length, width, and time, and the target image is processed into two-dimensional features including length and width. Based on the number of image frames N in the target video, the two-dimensional features corresponding to the target image are copied into N copies so that the video features and image features have the same feature dimension before fusion processing is performed. This process activates the parts of the target video that are related to the products that need attention in the target image, while ignoring the irrelevant parts. Here, N is a natural number greater than 1. The fused feature information is decoded using an attention-based deep learning model to determine the feature information of at least one candidate product identified from the target video; wherein, a random vector of a target number is used as the first input information of the deep learning model, and the feature information obtained after the fusion process is used as the second and third input information of the deep learning model. By matching the feature information of the at least one candidate product with the image features of key content in the target image, a target product with a confidence level that meets the conditions is determined from the at least one candidate product; The target product is segmented and tracked, and the product content segmentation result is output. The target video is edited based on the product content segmentation results to generate an explanatory video about the target product.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 8.
10. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 8.