Video object segmentation method, device, storage medium and program product

Abstract

The application provides a video target segmentation method, device, storage medium and program product. The method comprises the following steps: determining a video to be processed and a target object to be segmented in the video; and processing each frame image to be processed in the video in sequence based on a cycle feature by using a pre-constructed target segmentation model to segment the target object from each frame image. The target segmentation model comprises a cycle feature updating model, which is used to update the cycle feature according to the image being processed in the video processing process. The application can realize dynamic updating of the cycle feature by using the model, can not only make the cycle feature fuse the information of multiple processed frame images, but also effectively avoid feature stacking. The size of the cycle feature can not gradually increase with the progress of the video, the memory occupation is reduced, the segmentation effect of a long video is avoided due to insufficient memory, and the cost of the video target segmentation algorithm is reduced.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video target segmentation method, device, storage medium, and program product. Background Technology

[0002] Video object segmentation can determine the region where the target object is located from each frame of a video, and it plays an important role in scenarios such as target tracking and video matting.

[0003] During video object segmentation, segmented image information can be stored to guide the segmentation of subsequent images. However, this approach has drawbacks. As video processing progresses, segmented image information accumulates in memory, consuming significant amounts of memory. Furthermore, for longer videos, limited memory may not be sufficient to support the infinitely growing feature storage, impacting segmentation performance. Summary of the Invention

[0004] The main objective of this application is to provide a video target segmentation method, device, storage medium, and program product to reduce the cost of target segmentation in video and improve the segmentation effect.

[0005] In a first aspect, embodiments of this application provide a video target segmentation method, including:

[0006] Determine the video to be processed and the target objects to be segmented in the video;

[0007] By using a pre-constructed target segmentation model, each frame of the video to be processed is processed sequentially based on cyclic features to segment the target object from each frame; wherein, the target segmentation model includes a cyclic feature update model, which is used to update the cyclic features according to the currently processed image during video processing.

[0008] Optionally, the target segmentation model further includes: an encoder, a feature reading module, and a decoder;

[0009] Specifically, when processing any frame of an image in a video, the encoder is used to extract the image features corresponding to that image;

[0010] The feature reading module is used to calculate the association information between the image features and the current loop features;

[0011] The decoder is used to obtain the target segmentation result of the image based on the associated information.

[0012] Optionally, the number of target objects is at least one, and the method further includes:

[0013] Based on the target segmentation results of the video, target videos for each target object are obtained; wherein, the target video of the target object is the video obtained after removing the background of the target object from the video to be processed, or the target video is composed of multiple frames of images in the video to be processed containing the target object;

[0014] Output the target video and / or at least a portion of the images from the target video for each target object.

[0015] Optionally, output the target video and / or at least a portion of the images from the target video for each target object, including:

[0016] Display the target video and / or at least a portion of the images in the target video;

[0017] Based on the user's editing operations on the target video and / or at least some of the images in the target video, a promotional video and / or promotional images of the target object are generated.

[0018] Optionally, the processing operations on the images in the video include:

[0019] The process involves acquiring the image to be processed from the video and the current loop features, and extracting the image features corresponding to the image; wherein the loop features are determined based on the processed images in the video.

[0020] Based on the image features and the loop features, the target segmentation result of the image to be processed is determined, and the image features and the loop features are input into the loop feature update model to obtain the updated loop features, so as to process the image to be processed in the next frame of the video according to the updated loop features.

[0021] Optionally, determining the video to be processed and the target object to be segmented in the video includes:

[0022] The video to be processed is acquired, and at least one reference image in the video and the target segmentation result corresponding to each reference image are determined.

[0023] The method further includes: determining initial cyclic features based on the at least one frame of reference image and the corresponding target segmentation result;

[0024] The target segmentation result of the reference image is determined by the user annotating the target object in the reference image, or by automatically annotating the reference image based on a preset image of the target object.

[0025] The images to be processed in the video are images other than the reference image; the initial cyclic features are used to segment the first frame of the image to be processed.

[0026] Optionally, the method further includes at least one of the following:

[0027] A reference image from the video is displayed so that the user can mark the location of the target object on the reference image, and the target segmentation result of the reference image is determined based on the user's markings;

[0028] Semantic segmentation is performed on the reference image in the video to determine at least one object in the reference image; the image regions corresponding to the at least one object are compared with preset images of target objects to obtain the target segmentation result of the reference image;

[0029] Semantic segmentation is performed on the reference image in the video to identify at least one object in the reference image; the image region corresponding to the at least one object is displayed, the target object selected by the user from the at least one object is obtained, and the target segmentation result is determined based on the image region corresponding to the target object.

[0030] Optionally, the image features and cyclic features are input into a cyclic feature update model to obtain updated cyclic features, including:

[0031] The image features and cyclic features are input into the extraction module to obtain the extraction result; wherein, the extraction module is used to fuse the image features and cyclic features;

[0032] The refining result is input into the enhancement module to obtain the enhancement result; wherein, the enhancement module is used to perform a pooling operation on the refining result;

[0033] The extraction result and the enhancement result are added together and then input into the compression module to obtain the updated loop feature of the preset length.

[0034] Optionally, the extraction module includes a downsampling layer, a convolutional layer, a matrix calculation unit, and a normalization layer; the image features and cyclic features are input into the extraction module to obtain the extraction result, including:

[0035] The image features and cyclic features are downsampled by a downsampling layer, and the downsampled image features and cyclic features are respectively input into a convolutional layer to obtain a first feature matrix corresponding to the image features and a second feature matrix corresponding to the cyclic features.

[0036] The matrix computation unit performs matrix correlation operations on the first and second feature matrices, and inputs the results into the normalization layer to obtain the refined results.

[0037] Secondly, embodiments of this application also provide a video target segmentation method, including:

[0038] Identify the video to be processed and the target products to be segmented within the video;

[0039] By using a pre-built target segmentation model, each frame of the video to be processed is processed sequentially based on cyclic features to segment the target product from each frame of the video; wherein, the target segmentation model includes a cyclic feature update model, which is used to update the cyclic features according to the currently processed image during video processing;

[0040] Based on the target segmentation results of the video, a target video and / or target image of the target product are generated.

[0041] Optionally, the target image is the image containing the target object in the video to be processed, or it is the image obtained by removing the background from the image containing the target object; the method further includes:

[0042] Obtain the filtering conditions input by the user, and filter the target image according to the filtering conditions;

[0043] The selected images are updated to the media library, which is used to create promotional videos and / or images for the target product.

[0044] Thirdly, embodiments of this application provide an electronic device, including:

[0045] At least one processor; and

[0046] A memory that is communicatively connected to the at least one processor;

[0047] The memory stores instructions that can be executed by the at least one processor to cause the electronic device to perform the method described in any of the above aspects.

[0048] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the above aspects.

[0049] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above aspects.

[0050] The video target segmentation method, device, storage medium, and program products provided in this application can determine the video to be processed and the target objects to be segmented in the video. By using a pre-constructed target segmentation model, each frame of the video to be processed is processed sequentially based on cyclic features to segment the target objects from each frame. The target segmentation model includes a cyclic feature update model, which updates the cyclic features according to the currently processed image during video processing. This allows for dynamic updating of cyclic features, enabling the cyclic features to be integrated with information from multiple processed frames and effectively avoiding feature stacking. The size of the cyclic features does not gradually increase as the video progresses, reducing memory usage and preventing long videos from being affected by insufficient memory, thus lowering the cost of the video target segmentation algorithm. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0052] Figure 1 This is a schematic diagram illustrating the application of target segmentation in an embodiment of this application;

[0053] Figure 2 An application scenario diagram provided for this application;

[0054] Figure 3 A flowchart illustrating a video target segmentation method provided in an embodiment of this application;

[0055] Figure 4 A schematic diagram illustrating the principle of determining target segmentation results provided in an embodiment of this application;

[0056] Figure 5 A schematic diagram illustrating the updating of loop features based on step size, provided as an embodiment of this application;

[0057] Figure 6 A flowchart illustrating the application of target segmentation results provided in an embodiment of this application;

[0058] Figure 7 A schematic diagram illustrating the principle of updating a loop feature, provided in an embodiment of this application;

[0059] Figure 8 A flowchart illustrating another video target segmentation method provided in this application embodiment;

[0060] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0061] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0063] First, let me explain the terms used in this application:

[0064] Semi-supervised video object segmentation (VOS) involves providing initial annotations for one or more objects to be segmented in the first frame of a video. In subsequent video frames, an algorithm segments the objects provided in the first frame. VOS requires minimal interaction to complete the segmentation of the entire video, reducing manpower and improving efficiency.

[0065] Neural network models: complex network systems formed by a large number of interconnected neurons, which have wide applications in fields such as image processing.

[0066] The embodiments of this application can be applied to any scenario requiring target segmentation in video. For example, in video editing, post-production, and video conferencing, it is often necessary to perform precise, pixel-level segmentation of targets in the video. Users can refine the target segmentation result of the first frame through interactive clicks, and the algorithm can segment the target object in all subsequent frames based on the video and the segmentation result of the target object to be segmented in the first frame.

[0067] Optionally, in the product category, the video can be segmented to extract the target product from the video. After removing the background, it can be used for material production, etc.

[0068] Figure 1 This is a schematic diagram illustrating an application of target segmentation provided in an embodiment of this application. For example... Figure 1 As shown, the target segmentation method can segment the target product, such as a cup, from each frame of a video. After obtaining the target segmentation result, a background can be added to the target product, and the video can be edited to obtain the processed video, such as an advertising video for the target product or other types of product videos, which can quickly realize the production of various types of product videos.

[0069] Optionally, embodiments of this application can be particularly applied to target segmentation of live video. For example, in a virtual character live streaming scenario, the virtual character can introduce a target product, forming a video stream showcasing the target product that is transmitted to the user's terminal for convenient viewing. Target segmentation can be performed on the real-time video stream during the live stream, or it can be performed after the live stream ends. Based on the target segmentation results, the image frames in which the target product appears and its position within each frame can be determined, thereby enabling the use of the live video to obtain video and image materials of the target product for subsequent use.

[0070] In the field of video conferencing, participants can be filmed using a camera to obtain a video stream. By using target segmentation methods, participants can be separated from the video stream, and the background can be blurred or a virtual background can be added to form a processed video stream that is displayed to other participants. This effectively protects user privacy and meets users' personalized needs.

[0071] In the short video field, target segmentation methods can be used to segment target objects in a video and use them as video production material. For example, after a user records a dance video indoors, the target person in the video can be segmented, and scenes such as pastoral scenes or party scenes can be added. This allows users to choose appropriate scenes based on dance style and movements, improving the production effect of the short video.

[0072] In the process of target segmentation in video, to achieve better segmentation results, the segmentation result for each frame can be obtained by utilizing that frame and historical information. The historical information refers to information from previously segmented images in the video. As the video progresses, the historical information continuously increases, affecting the video segmentation effect.

[0073] In view of this, this application provides a video target segmentation method that can set cyclic features to assist in target segmentation. During the video processing, the cyclic features are used to process the current image to obtain the target segmentation result of the current image. The cyclic features and the current image features are then fused using a model to obtain updated cyclic features for subsequent image processing.

[0074] Figure 2 This is an application scenario diagram provided for this application. For example... Figure 2As shown, the user terminal can upload captured videos containing the target object to the server. The server can perform target segmentation on the video based on a target segmentation model, which may include a cyclic feature update model and a segmentation module. When processing each frame, the current cyclic features are retrieved from the storage unit. The image features and the current cyclic features are input into the segmentation module to obtain the target segmentation result for that image. The image features and cyclic features are then input into the cyclic feature update model to obtain the updated cyclic features, which are then updated in the storage unit. The updated cyclic features can be used when processing the next frame. After processing all frames in the video, the server can feed back the target segmentation result of the video to the user terminal.

[0075] By continuously fusing the currently processed image features and cyclic features through the cyclic feature update model, updated cyclic features are obtained. This enables the cyclic features to dynamically change, containing information from multiple processed frames while effectively avoiding feature stacking. The size of the cyclic features is related to the output dimension of the model and does not gradually increase as the video progresses, reducing memory usage and preventing long videos from being affected by insufficient memory, thus lowering the cost of video object segmentation algorithms and improving object segmentation results.

[0076] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0077] Figure 3 This is a flowchart illustrating a video target segmentation method provided in an embodiment of this application. In this embodiment, the executing entity of the method can be any device with image processing capabilities, such as a server, user terminal, etc. Figure 3 As shown, the method may include:

[0078] Step 301: Determine the video to be processed and the target object to be segmented in the video.

[0079] The video can be of any type, such as a long video or a short video. The video may contain at least one object, and embodiments of this application can be used to segment a target object from the video.

[0080] Optionally, the target object can be set according to actual needs, such as a target product or a person. The number of target objects can be one or more. The target object can be determined by the user or automatically by the device.

[0081] Step 302: Using a pre-constructed target segmentation model, process each frame of the video to be processed sequentially based on cyclic features to segment the target object from each frame; wherein, the target segmentation model includes a cyclic feature update model, which is used to update the cyclic features according to the currently processed image during video processing.

[0082] The video may include multiple frames of images. When segmenting the video, each frame of the video can be processed sequentially according to the cyclic feature.

[0083] Optionally, the cyclic feature stores features of processed images in the video, which can be used as historical features for segmenting each frame to assist in achieving target segmentation of the current image. Furthermore, the cyclic feature is continuously updated during the processing of multiple frames of the video. The cyclic feature can be stored in memory or other storage devices.

[0084] Optionally, the cyclic feature can specifically be a Recurrent Dynamic Embedding (RDE), which is an embedding feature that is continuously updated and dynamically changed during the processing of multiple frames of video images. The cyclic feature can be obtained through a cyclic feature update model. The cyclic feature update model can be a machine learning model, such as a neural network model.

[0085] When processing any frame of an image, the current cyclic features can be obtained. The image can be processed using the current cyclic features to obtain the target segmentation result. The target segmentation result can be used to represent the location of the target object in order to segment the target object from the image.

[0086] In one example, the target segmentation result may include annotations for each pixel in the image, which indicate whether the pixel belongs to the target object. For example, the annotation numbers can take values ​​between 0 and 1. If any pixel in the image is labeled 0, it means that the pixel does not belong to the target object; if the pixel is labeled 1, it means that the pixel belongs to the target object. By labeling each pixel in the image with 0s and 1s, the pixels contained in the target object can be determined, thereby achieving pixel-level segmentation of the target object.

[0087] In another example, the target segmentation result can be used to identify multiple target objects in an image, with different numbers representing different target objects. For example, when labeling pixels, the numbers 1, 2, and 3 represent target object 1, target object 2, and target object 3, respectively, while the number 0 represents the background region of no interest.

[0088] In this embodiment, after reading the cyclic features from the storage space, updating the cyclic features and determining the target segmentation result can be performed simultaneously or sequentially; this embodiment does not restrict the execution order of the two. It should be noted that, regardless of the order, the cyclic features before the update can be used to determine the target segmentation result.

[0089] In summary, the video target segmentation method provided in this embodiment can determine the video to be processed and the target objects to be segmented in the video. By using a pre-constructed target segmentation model, the method processes each frame of the video to be processed sequentially based on cyclic features, and segments the target objects from each frame. The target segmentation model includes a cyclic feature update model, which updates the cyclic features according to the currently processed image during video processing. This allows for dynamic updating of cyclic features through the model, enabling the cyclic features to be integrated with information from multiple processed frames and effectively avoiding feature stacking. The size of the cyclic features does not gradually increase as the video progresses, reducing memory usage and preventing long videos from being affected by insufficient memory, thus lowering the cost of the video target segmentation algorithm.

[0090] In one or more embodiments of this application, optionally, the processing operation of the images in the video may include: acquiring the image to be processed in the video and the current loop feature, and extracting the image feature corresponding to the image; wherein, the loop feature is a feature determined based on the processed images in the video; determining the target segmentation result of the image to be processed based on the image feature and the loop feature, and inputting the image feature and the loop feature into the loop feature update model to obtain the updated loop feature, so as to process the image to be processed in the next frame of the video based on the updated loop feature.

[0091] Specifically, when processing any frame of an image, corresponding image features can be obtained from that image. Optionally, these image features can be extracted using an encoder. After obtaining the image features, the image features and cyclic features can be input into a deep learning model to obtain the corresponding target segmentation result. Alternatively, matrix operations or other operations can be performed on the image features and cyclic features to obtain the corresponding target segmentation result. Or, the deep learning model can be combined with matrix operations or other operations to comprehensively determine the target segmentation result.

[0092] In addition to determining the target segmentation result, image features can also be used to update cyclic features. Optionally, the image features and the current cyclic features can be input into the cyclic feature update model to obtain the updated cyclic features.

[0093] In practical applications, images in a video can be segmented sequentially. When processing each frame, the target segmentation result of the current image can be determined based on the current image's features and cyclic features, and the cyclic features can be updated. When processing the next frame, the updated cyclic features are used.

[0094] In other alternative implementations, the target segmentation algorithm described above can be applied only to a subset of the images in the video. For example, the target segmentation algorithm can be used to process only a subset of the images in the video to obtain the target segmentation results, while the target segmentation results for the remaining images can be obtained by user input or other methods. Alternatively, the remaining images may not be subject to target segmentation.

[0095] In summary, the video target segmentation method provided in this embodiment can acquire the image to be processed in the video and the current loop features, and extract the image features corresponding to the image. The loop features are features determined based on the processed images in the video. Based on the image features and the loop features, the target segmentation result of the image to be processed is determined, and the image features and loop features are input into the loop feature update model to obtain the updated loop features. The next frame of the image to be processed in the video is then processed based on the updated loop features. Thus, during the video processing, the loop features can be continuously updated by the model using existing image features, enabling the loop features to integrate information from the processed images, quickly and accurately updating the loop features, and improving the efficiency and accuracy of loop feature updates.

[0096] In other alternative implementations, the image and cyclic features can be input into the cyclic feature update model to obtain the updated cyclic features.

[0097] In one or more embodiments of this application, optionally, the target segmentation model further includes: an encoder, a feature reading module, and a decoder; wherein, when processing any frame of an image in a video, the encoder is used to extract the image features corresponding to the image; the feature reading module is used to calculate the association information between the image features and the current cyclic features; and the decoder is used to obtain the target segmentation result of the image based on the association information.

[0098] Accordingly, determining the target segmentation result of the image frame to be processed based on the image features and cyclic features may include: inputting the image features and cyclic features into the feature reading module to obtain the association information between the image features and cyclic features; and inputting the association information into the decoder to obtain the target segmentation result of the image to be processed.

[0099] Figure 4This is a schematic diagram illustrating the principle of determining target segmentation results, provided in an embodiment of this application. Figure 4 As shown, each frame of the image is input into the encoder to obtain the corresponding image features. The image features and the loop features are input together into the feature reading module for calculation. The calculation result is then input into the decoder to obtain the target segmentation result.

[0100] Optionally, the input and output sizes of the image and each model can be set according to actual needs. For example, the image to be processed can be an RGB image containing H*W*3 data, which is input into the encoder to obtain h*w*c image features. The size of the cyclic features can be the same as the image features. Both are input into the feature reading module, calculated, and then input into the decoder to obtain the H*W target segmentation result.

[0101] The feature reading module can be used to fuse image features and cyclic features to obtain associated information. Optionally, both image features and cyclic features can be represented in matrix form, and the feature reading module can perform matrix multiplication and / or matrix addition operations on image features and cyclic features to achieve their fusion.

[0102] based on Figure 4 The scheme shown maintains historical information through a cyclic feature. Historical image frames are encoded to obtain image features, which are then used as input along with the original cyclic features. The model is updated using this cyclic feature to obtain new cyclic features. When the current frame image needs to be segmented, features are encoded and input along with the cyclic features into a feature reading module. Finally, the segmentation result is obtained through a decoder.

[0103] In summary, by using an encoder, a feature reading module, and a decoder, image features can be extracted, image features and cyclic features can be fused, and the fused features can be decoded to obtain the target segmentation result. This can effectively utilize cyclic features to guide image segmentation, thereby improving the efficiency and accuracy of image segmentation.

[0104] In one optional implementation, a certain step size can be set, and the cyclic feature is updated after the step size requirement is met. Optionally, inputting the image features and cyclic features into the cyclic feature update model to obtain the updated cyclic features may include: determining whether the current counter is equal to the step size; if it is equal, then inputting the image features and cyclic features into the cyclic feature update model to obtain the updated cyclic features, and resetting the counter to 0. If it is not equal, then the counter can be incremented by 1, and the update of the cyclic feature can be skipped.

[0105] Figure 5 This is a schematic diagram illustrating an embodiment of updating loop features based on a step size. Figure 5As shown, the step size can be set to θ. The cyclic feature update model can be specifically SAM (Spatio-temporal Aggregation Module), where the spatio-temporal aggregation module can aggregate image features and cyclic features in the spatio-temporal domain.

[0106] Optionally, the image features of frame t-nθ and the corresponding cyclic features of frame t-nθ can be input into the SAM to obtain updated cyclic features. As video processing progresses, the image features of frame t-θ and the corresponding cyclic features of frame t-θ can be input into the SAM to obtain the cyclic features corresponding to frame t. The cyclic features corresponding to frame t can be the cyclic features used when determining the target segmentation result of frame t.

[0107] For example, initial cyclic features can be determined based on the image features of the first frame and the user-annotated target segmentation results. The target segmentation results of the second frame are then determined based on the image features of the second frame and the initial cyclic features.

[0108] Assuming a step size of 3, since the second frame does not yet meet the step size requirement, the cyclic features are not updated. The third frame image uses the current cyclic features, i.e., the initial cyclic features, together with the image features of the third frame to obtain the target segmentation result of the third frame image, and the cyclic features are not updated.

[0109] When processing the fourth frame image, since the step size requirement is met, the target segmentation result of the fourth frame image can be determined based on the initial cyclic features and the image features of the fourth frame, and the cyclic features are updated based on the image features of the fourth frame. The updated cyclic features can be used to process the fifth frame image, and together with the image features of the fifth frame, the target segmentation result of the fifth frame image is obtained.

[0110] In summary, the update frequency of cyclic features can be adjusted by changing the step size, allowing the target segmentation algorithm to balance efficiency and accuracy. The step size can also be input by the user; in scenarios with high real-time requirements, a larger step size can be set, while a smaller step size can be set for less demanding applications. This allows users to adjust the update frequency of cyclic features according to the scenario, meeting the needs of different situations and improving the algorithm's flexibility.

[0111] Figure 6 This is a flowchart illustrating the application of a target segmentation result provided in an embodiment of this application. For example... Figure 6 As shown, after obtaining the target segmentation result, the following operations can also be performed.

[0112] Step 601: Based on the target segmentation results of the video, obtain the target video of each target object; wherein, the target video of the target object is the video obtained after removing the background of the target object from the video to be processed, or the target video is composed of multiple frames of images of the target object in the video to be processed.

[0113] The number of target objects to be segmented can be at least one. Each target object can have a corresponding target segmentation result. Based on the target segmentation result, the target video can be extracted, and the target video can include multiple frames of images.

[0114] In one example, the extracted multi-frame images can be images containing the target object from the original video. For instance, if the original video has 1000 frames and the target object appears in 600 frames, these 600 frames can be extracted to form the target video. If the multi-frame images containing the target object are consecutive, a single target video can be formed; if they are not consecutive, one or more target videos can be formed. For example, if the target object appears in frames 1 to 200, one target video can be formed; if the target object disappears after 200 frames and reappears in frames 301 to 700, another target video can be formed.

[0115] By extracting the images containing each target object, the target video of each target object can be quickly and accurately extracted from the original video, thus realizing object-based video segmentation.

[0116] In another example, the original video can be cut out based on the target segmentation result to obtain the target video after removing the background. Only the target object appears in the target video, and no other objects appear. This can quickly achieve video cutout operation for the target object.

[0117] Step 602: Output the target video and / or at least some images from the target video for each target object.

[0118] Optionally, the target video and / or at least some images from the target video can be displayed to the user for viewing or editing, and the user can also be provided with the ability to download the target video and / or at least some images from the target video. Alternatively, the target video and / or at least some images from the target video can be uploaded to the cloud for storage.

[0119] In one optional implementation, outputting the target video and / or at least some images in the target video for each target object may include: displaying the target video and / or at least some images in the target video; and generating promotional videos and / or promotional images for the target objects based on user editing operations on the target video and / or at least some images in the target video.

[0120] Optionally, the target video can be played directly to the user, or each frame of the target video can be statically displayed to the user. The user can perform operations such as editing, resizing, beautifying, and adding text to the target video or images to generate promotional videos or images for the target object.

[0121] For example, in the product category, after determining the image frame containing the target product in the original video using a target segmentation algorithm, the image frame can be extracted to form the target video. Alternatively, the original video can be processed by image matting to obtain the target video. The target video can be used as a promotional video or image for the target product, such as an advertising video, poster, etc., and published on e-commerce platforms or other platforms. Alternatively, the target video can be in the form of a short video and published on a short video platform. Furthermore, users can edit the target video or image and publish the edited video.

[0122] By displaying the target video or at least some images in the target video, and generating promotional videos or images of the target object based on the user's editing operations on the target video or the at least some images, the algorithm can assist users in creating promotional materials for the target object, improve the efficiency of material production, and enhance the user experience.

[0123] In another alternative implementation, after obtaining the target video, the target video or at least some of its images can be added to the media library, or added to the media library after being filtered or edited by the user.

[0124] Optionally, user-input filtering criteria can be obtained, and target videos and / or target images can be filtered according to the filtering criteria. The filtered videos or images can then be updated to a media library, which can be used to create promotional videos and / or promotional images for the target object. Here, a target image can refer to an image from a target video.

[0125] Optionally, the filtering conditions can be set according to actual needs. For example, they may include, but are not limited to: images of the target object in its entirety, images of the target object displayed from the front, images of the target object in a certain special form, and video clips of the target object in a certain changing state.

[0126] For example, in the product category, the filtering criteria can be: product introduction images, images showing the product in its entirety, images with the product fully packaged, images with the packaging removed, images with some packaging removed, video clips showing the product in use, unboxing videos, etc. Users can select from target videos or images based on the filtering criteria input by the user. During selection, methods such as simple comparison or machine learning models can be used to determine which images meet the filtering criteria.

[0127] Alternatively, after obtaining the target video or image, the user can edit the target video or image and store the edited video or image in the media library.

[0128] Optionally, the target object's identifier, attribute information, etc., can be stored together with the edited or filtered videos and images in the media library. In practical applications, users can enter search criteria, such as the target object's identifier, to quickly find the corresponding videos or images for use in creating short videos, posters, promotional materials, advertising videos, etc., for the target object.

[0129] In summary, by performing image matting or image extraction operations on the original video based on the target segmentation results of the video, the target videos of each target object are obtained, and the target videos of each target object or at least some images in the target videos are output. This can improve the production efficiency and accuracy of target object videos or images, and meet the usage needs in different scenarios.

[0130] In one or more embodiments of this application, optionally, determining the video to be processed and the target object to be segmented in the video includes: acquiring the video to be processed, and determining at least one reference image in the video and the target segmentation result corresponding to each reference image.

[0131] Correspondingly, initial cyclic features can also be determined based on the at least one reference image and the corresponding target segmentation results.

[0132] Optionally, the video can be acquired through various methods, such as being captured by a user, reading local video, or obtaining it from a server. The reference image is used to determine the initial loop characteristics, and the reference image can be any one or more frames from the video, for example, it can be the first frame of the video.

[0133] The target segmentation result of the reference image is determined by the user annotating the target object in the reference image, or by automatically annotating the reference image based on a preset image of the target object. The image to be processed is any image in the video other than the reference image, and the corresponding target segmentation result is used to segment the target object from the image; the initial cyclic features are used to segment the first frame of the image to be processed.

[0134] In an alternative implementation, a semi-supervised video object segmentation method can be used, whereby, after acquiring the video, the location of the target object can be determined by having the user annotate the baseline image.

[0135] In another optional implementation, the reference image can be automatically labeled based on a preset image of the target object. The preset image of the target object can be an image other than the video, and more particularly, a typical image capable of accurately displaying the target object. For example, for a target product, the preset image could be the main image of the target product on the product details page.

[0136] After determining the target segmentation result of the reference image, the initial cyclic features can be determined based on the reference image and the target segmentation result, and the cyclic features can be used to guide the segmentation of other images.

[0137] In practical applications, the user or a preset image can first annotate the first frame of the video to determine the target segmentation result of the first frame. Based on the first frame and the target segmentation result, an initial cyclic feature is generated. Then, the second frame is processed, and the target segmentation result of the second frame is determined according to the method provided in the aforementioned embodiment. The cyclic feature is then updated, and the third frame is processed based on the updated cyclic feature. This process continues until all images in the video have been processed.

[0138] Alternatively, the user or a preset image can be used to annotate the last frame of the video, and the algorithm can then segment the remaining frames sequentially from the end to the beginning. Alternatively, the middle frames of the video can be annotated, and the algorithm can segment the images sequentially from the middle frames forward and backward.

[0139] In summary, by first having the user or based on preset images to annotate at least one frame of a reference image in the video, the target segmentation result of the reference image is determined. Based on at least one frame of the reference image and the target segmentation result corresponding to each reference image, the initial cyclic features are determined. This allows the initial cyclic features to reference more accurate target segmentation results, guide the prediction of target segmentation results for subsequent images, and improve the segmentation accuracy of subsequent images.

[0140] In one or more embodiments of this application, optionally, the following annotation methods can be used to annotate the reference image.

[0141] In the first optional annotation method, a reference image in the video can be displayed so that the user can annotate the location of the target object on the reference image, and the target segmentation result of the reference image can be determined based on the user's annotation.

[0142] In one example, after a reference image is shown to the user, the user can draw the outline of the target object in the reference image, thereby determining the precise location of the target object in the reference image.

[0143] In another example, a user can use image editing software to annotate target objects in a reference image, and the algorithm can obtain the location of the target objects from the target editing software.

[0144] In this implementation, the user annotates the baseline image, which improves the accuracy of the annotation and thus enhances the accuracy of video target segmentation.

[0145] In the second optional annotation method, semantic segmentation can be performed on the reference image in the video to determine at least one object in the reference image; the image regions corresponding to the at least one object are compared with preset images of the target objects to obtain the target segmentation result of the reference image.

[0146] For example, the similarity between the image region corresponding to each object and a preset image can be determined, wherein the object with the highest similarity can be determined as the target object in the reference image.

[0147] Optionally, after identifying the object with the highest similarity, the region where the object is located can be highlighted and displayed to the user, who can then confirm whether the object belongs to the target object. After the user's confirmation, the target segmentation result of the baseline image can be generated.

[0148] In this implementation, the baseline image is annotated using a preset image of the target object, which reduces the user's workload and improves the user experience.

[0149] In the third optional annotation method, semantic segmentation can be performed on the reference image in the video to determine at least one object in the reference image; the image region corresponding to the at least one object can be displayed, the target object selected by the user from the at least one object can be obtained, and the target segmentation result can be determined according to the image region corresponding to the target object.

[0150] For example, different colors can be set for the image areas containing different objects, or the outlines of each object can be drawn with lines. Users can directly click on the image area containing a certain object, which can be used as the target object. In addition, users can also adjust the outline of the target object.

[0151] In this implementation, users can directly annotate the benchmark image by clicking, without having to draw the outline of the benchmark image, thus improving the annotation efficiency of the benchmark image and balancing annotation accuracy and user experience.

[0152] The above annotation methods can be used individually or in combination. For example, the first method can be used for part of the reference image, and the second method can be used for the rest of the reference image. Alternatively, for the same reference image, the second and third methods can be used for annotation, and the results of the two methods can be fused to determine the final target segmentation result.

[0153] In one or more embodiments of this application, optionally, the following operations may be performed for each image in the video: semantic segmentation of the image to determine at least one object in the image, and comparison of the image region corresponding to each object with a preset image of the target object to determine the similarity between each object and the target object, and taking the highest similarity as the similarity of the image.

[0154] From each frame of the video, one or more images with the highest similarity are selected as reference images, and the target segmentation result is determined based on the object with the highest similarity to the target object in the reference image.

[0155] For example, if the first frame contains three objects with similarities of 10%, 1%, and 80% to the target object, then the highest similarity, 80%, can be taken as the similarity between the first frame and the target object. Similarly, the similarity between the other frames and the target object can be determined. Assuming that the highest similarity of a certain frame is 90%, then that frame can be used as the reference image.

[0156] In summary, by comparing each frame of the video with the target object and determining the image with the highest similarity as the baseline image, the accuracy of initial annotation can be improved without relying on the user.

[0157] In one or more embodiments of this application, optionally, in scenarios where initial annotation relies on the user, the similarity between each frame of the video and the previous frame can be calculated. If the similarity between two adjacent frames is lower than a preset similarity threshold, the next frame of the two adjacent frames can be used as the reference image and displayed to the user for annotation. Alternatively, one or more sets of adjacent images with the lowest similarity can be selected from the video, and the next frame of the adjacent images can be used as the reference image and displayed to the user for annotation.

[0158] In summary, when users annotate the initial image, an image with low similarity to the previous frame can be selected from the video as the reference image. Since the reference image changes significantly from the previous frame, the segmentation effect of the reference image by the algorithm is slightly worse than the segmentation effect of other images by the algorithm. User annotation of the reference image can reduce the situation where the target segmentation effect is poor due to sudden changes in the video scene, and further improve the accuracy of video target segmentation.

[0159] In one or more embodiments of this application, optionally, determining the initial cyclic features based on the at least one frame of reference image and the corresponding target segmentation result includes: stitching the at least one frame of reference image and the corresponding target segmentation result together, and inputting the stitching result into an encoder to obtain the initial cyclic features.

[0160] For example, when there is only one reference image, the reference image and the corresponding target segmentation result can be concatenated and input into the encoder to obtain the initial cyclic features. When there are multiple reference images, the reference image and the target segmentation result of each frame can be concatenated sequentially and input into the encoder to obtain the initial cyclic features.

[0161] Optionally, the encoder can be an encoder with a fixed input dimension or an encoder that can accept different input dimensions, so as to determine an initial cyclic feature of a fixed size based on a non-fixed number of reference images.

[0162] The encoder can be implemented using convolutional neural networks or similar methods, and can be trained together with a recurrent feature update model, a segmentation module, or similar components.

[0163] The encoder for extracting image features and the encoder for determining the initial recurrent features can be different. Optionally, the encoder for extracting image features can be referred to as the first encoder, and the encoder for determining the initial recurrent features can be referred to as the second encoder. The first encoder and the second encoder have different functions, but both can be trained together with other modules.

[0164] In summary, by stitching together at least one frame of the reference image and the corresponding target segmentation result, and inputting the stitched result into the encoder, features of the reference image and the target segmentation result can be extracted. Based on the extracted features, initial loop features can be determined, which enables the loop features to retain key information of the reference image, improves the accuracy of the initial loop features, and enhances the overall segmentation effect of the video.

[0165] In other alternative implementations, when there are multiple frames of reference images, one frame can be selected to determine the initial cyclic features. The remaining reference images can be used directly instead of calculating the initial cyclic features. During the video target segmentation process, these reference images can be skipped, thus improving the efficiency of video target segmentation.

[0166] Figure 7 This is a schematic diagram illustrating the principle of updating a loop feature, as provided in an embodiment of this application. Figure 7As shown, the cyclic feature update module may include: an extraction module, an enhancement module, and a squeezing module. Inputting the image features and cyclic features into the cyclic feature update model to obtain updated cyclic features may include: inputting the image features and cyclic features into the extraction module to obtain an extraction result; wherein the extraction module is used to fuse the image features and cyclic features; inputting the extraction result into the enhancement module to obtain an enhancement result; wherein the enhancement module is used to perform pooling operations on the extraction result; adding the extraction result and the enhancement result and then inputting them into the squeezing module to obtain updated cyclic features of a preset length.

[0167] Optionally, the cyclic feature can be a feature of a preset length, and the preset length is maintained during video target segmentation. Specifically, when updating the cyclic feature, the image feature and the current cyclic feature can be concatenated and input into the extraction module. The extraction module can fuse the input features, and then the enhancement module can enhance the features. Finally, the extracted features and the enhanced features are added together and compressed to a preset length by the compression module to obtain the updated cyclic feature.

[0168] In summary, in video target segmentation, by using extraction, enhancement, and compression modules, lightweight and constant-size cyclic features can be generated as historical information for storage and to guide target segmentation. Regardless of the number of frames processed, the size of the cyclic features remains unchanged, thus meeting the lightweight requirement.

[0169] In one or more embodiments of this application, optionally, the extraction module can be used to perform multiplication or addition operations on the features of the image and the features of the cycle features to achieve the fusion of image features and cycle features.

[0170] Optionally, the extraction module may include a downsampling layer, a convolutional layer, a matrix calculation unit, and a normalization layer. Inputting the image features and cyclic features into the extraction module to obtain the extraction result includes: downsampling the image features and cyclic features through the downsampling layer, and inputting the downsampled image features and cyclic features into the convolutional layer respectively to obtain a first feature matrix corresponding to the image features and a second feature matrix corresponding to the cyclic features; performing matrix correlation operations on the first feature matrix and the second feature matrix through the matrix calculation unit, and inputting the operation result into the normalization layer to obtain the extraction result.

[0171] Specifically, after downsampling the image features and cyclic features, they are input into the convolutional layer. The convolutional layer can extract the matrix corresponding to the image features and the matrix corresponding to the cyclic features. These two matrices represent deeper features of the image features and cyclic features. By performing matrix-related operations on the two matrices, such as matrix addition and / or matrix multiplication, the image features and cyclic features can be fused. Then, a normalization operation is performed to obtain the refined result.

[0172] In one implementation, the enhancement module can specifically be used to perform pyramid pooling on the refining results. Optionally, the enhancement module may include a pyramid pooling layer, such as ASPP (atrous spatial pyramid pooling), which can better perform pyramid pooling on the refining results to obtain a fixed-size enhanced result.

[0173] In summary, the extraction module enhances feature correlation in the spatiotemporal domain through matrix correlation calculations, enabling better fusion of image features and cyclic features. The enhancement module, through ASPP, can acquire receptive fields of different scales and has different feature filtering capabilities in space and time, thereby further improving the target segmentation effect of the video.

[0174] In one or more embodiments of this application, optionally, the model can be trained first using training samples, and the trained model can be used to segment the target in the video to be processed.

[0175] In one optional implementation, training samples can be obtained, including a training video and target segmentation results corresponding to each image in the training video. During training, at least one reference image is selected from the training video, and the other images in the training video besides the at least one reference image are images to be processed; the target segmentation result of the at least one reference image is perturbed to obtain a noisy target segmentation result, and initial cyclic features are determined based on the noisy target segmentation result. For any image to be processed in the training video, the following operations are performed: predict the target segmentation result of the image based on the current cyclic features, and update the parameters of the cyclic feature update model according to the prediction result and the target segmentation results in the training samples; perturb the image features corresponding to the image to obtain noisy image features, and update the cyclic features according to the noisy image features, so as to process the next image to be processed in the training video according to the updated cyclic features.

[0176] Optionally, during training, the loss value can be determined by comparing the predicted and actual results for the images to be processed. The model parameters can then be updated based on the loss value to make the predicted results closer to the actual results. Here, the predicted result refers to the target segmentation result obtained by the algorithm, while the actual result is the target segmentation result corresponding to the images in the training samples.

[0177] To improve the algorithm's robustness to noise, perturbations can be added during training to update cyclic features. Taking the first frame as the baseline image, the target segmentation result of the first frame can be perturbed to obtain a noisy target segmentation result and construct initial cyclic features. When processing subsequent images, the image features can be perturbed, and the cyclic features can be updated based on the noisy image features.

[0178] Optionally, the step size can be set to 2, in which case no perturbation is added when processing frames 2, 4, and 6, and perturbation is added and the loop feature is updated when processing frames 3, 5, and 7.

[0179] When using the model, the video to be processed is continuously segmented. Since the historical segmentation results may not be accurate and may introduce a lot of noise, resulting in poor video segmentation results, during training, by constructing interference inputs, we can ensure that the model learns to overcome the noise problem in feature storage, making the model more robust to noise. In practical applications, this overcomes the noise brought by video sequences and improves the effect and performance of target segmentation.

[0180] Optionally, the various modules used in the target segmentation algorithm, such as the first encoder, second encoder, decoder, and recurrent feature update model, can be trained together.

[0181] In other alternative implementations, the structure of the above model can be adjusted according to actual needs, adding, removing, or replacing certain modules. For example, the feature reading module and the decoder can be merged into one module, and the recurrent feature update model can be a convolutional neural network model with other structures, etc.

[0182] Figure 8 This is a flowchart illustrating another video target segmentation method provided in an embodiment of this application. Figure 8 As shown, the method may include:

[0183] Step 801: Determine the video to be processed and the target product to be segmented in the video.

[0184] Optionally, the video to be processed can be a video corresponding to the target product. The video corresponding to the target product can be filmed by the user, or it can be a video of the target product obtained from other devices, such as a video of the target product on the product details page obtained from a server.

[0185] The product details page can be a page provided by an e-commerce platform to display products. The page can display the product's name, price, specifications, reviews, details, and can also display a video corresponding to the product.

[0186] Step 802: Using a pre-constructed target segmentation model, process each frame of the video to be processed sequentially based on cyclic features to segment the target product from each frame of the video; wherein, the target segmentation model includes a cyclic feature update model, which is used to update the cyclic features according to the currently processed image during video processing.

[0187] Optionally, for each image to be processed in the video, the following processing can be performed sequentially: based on the image features and cyclic features corresponding to the current image to be processed, the target segmentation result of the image is determined, and the image features and cyclic features are input into the cyclic feature update model to obtain the updated cyclic features, so as to process the next image to be processed in the video according to the updated cyclic features; wherein, the target segmentation result is used to characterize the position of the target product in the image.

[0188] Optionally, referring to any of the foregoing embodiments, the target object can be replaced with the target product, and the images in the video can be processed to obtain the target segmentation results of each image.

[0189] Step 803: Based on the target segmentation results of the video, generate the target video and / or target image of the target product.

[0190] Optionally, the target image can be the image containing the target object in the video to be processed, or the image obtained after removing the background from the image containing the target object; the target video can be a video composed of the target images.

[0191] Optionally, the target product can be extracted from each image based on the target segmentation results to form a target image or target video. This image or video does not contain the background of the original video and can be provided to merchants as video production material for editing. Alternatively, images or video clips containing the target product can be extracted from the original video and used as target images or target videos.

[0192] Optionally, the system can also obtain user-input filtering conditions, filter the target image according to the filtering conditions, and update the filtered images to the material library, which is used to create promotional videos and / or promotional images for the target product.

[0193] By filtering the generated target images based on user needs and using the filtered images as promotional material for products, product materials can be accurately extracted from the original video to meet users' personalized needs.

[0194] In an optional implementation, a reference image in the video can also be displayed for the user to mark the location of the target product on the reference image, and the target segmentation result of the reference image can be determined based on the user's markings; and / or, the main image corresponding to the target product on the product details page can be obtained, and semantic segmentation can be performed on the reference image in the video to determine at least one object in the reference image, and the image regions corresponding to the at least one object can be compared with the main image of the target product to obtain the target segmentation result of the reference image; wherein, the reference image and the corresponding target segmentation result are used to determine the initial cyclic features.

[0195] For example, the first frame image can be used as a reference image, and the user can mark the location of the target product in the first frame image. Alternatively, the main image of the target product can be obtained from the product details page, and the main image can be compared with each object in the first frame image to determine the target product, thereby obtaining the target segmentation result, which is used to construct the initial cyclic features.

[0196] In summary, the video target segmentation method provided in this embodiment can determine the video to be processed and the target product to be segmented in the video. Using a pre-constructed target segmentation model, it sequentially processes each frame of the video based on cyclic features, segmenting the target product from each frame. The target segmentation model includes a cyclic feature update model, used to update the cyclic features based on the currently processed image during video processing. Based on the target segmentation results of the video, it generates the target video and / or target image of the target product. Since the cyclic features do not gradually increase with the progress of the video, it effectively reduces memory usage, avoids long videos from being affected by insufficient memory in segmentation, lowers the cost of the video target segmentation algorithm, improves the target segmentation effect of product videos, and enhances the production efficiency and accuracy of product videos or images.

[0197] In the field of video conferencing, embodiments of this application also provide a video target segmentation method, including:

[0198] Identify the meeting videos to be processed and the target participants;

[0199] Using a pre-built target segmentation model, each frame of the meeting video is processed sequentially based on cyclic features to segment the target attendees from each frame. The target segmentation model includes a cyclic feature update model, which updates the cyclic features based on the currently processed image during video processing.

[0200] Based on the target segmentation results, the background of the meeting video is blurred or a virtual background is added.

[0201] For example, the device or server used by the target participant in a video conference can send the video, with background blurred or a virtual background added, to the devices used by other participants for playback. This effectively hides the user's environment, improves meeting efficiency, and protects user privacy. Optionally, the virtual background can be set by the user to meet their personalized needs.

[0202] In the field of short videos, embodiments of this application also provide a video target segmentation method, including:

[0203] Identify the long video to be processed and the target individuals to be segmented within the video;

[0204] Using a pre-built target segmentation model, the target person is segmented from each frame of the long video by sequentially processing the cyclic features based on the cyclic features. The target segmentation model includes a cyclic feature update model, which updates the cyclic features based on the currently processed image during video processing.

[0205] Based on the target segmentation results, a short video corresponding to the target person is generated.

[0206] The target person can be a user-specified person or a person who meets certain requirements, such as a person performing certain actions in a video.

[0207] For example, in a dance video, the dancers can be separated from the video and used as material to create short videos or other types of videos, which can effectively improve the production quality of short videos.

[0208] In the various embodiments of this application, the executing entity of the method can be set according to actual needs. For example, it can be executed by a terminal device, or it can be executed by a server, or some steps can be executed by a terminal device and some steps can be executed by a server.

[0209] Corresponding to the above-described video target segmentation method, this application embodiment also provides a video target segmentation apparatus, which may include:

[0210] The determination module is used to determine the video to be processed and the target objects to be segmented in the video.

[0211] The processing module is used to process each frame of the video to be processed sequentially based on cyclic features using a pre-built target segmentation model, and to segment the target object from each frame of the video; wherein, the target segmentation model includes a cyclic feature update model, which is used to update the cyclic features according to the currently processed image during the video processing.

[0212] In one or more embodiments of this application, optionally, the target segmentation model further includes: an encoder, a feature reading module, and a decoder;

[0213] Specifically, when processing any frame of an image in a video, the encoder is used to extract the image features corresponding to that image;

[0214] The feature reading module is used to calculate the association information between the image features and the current loop features;

[0215] The decoder is used to obtain the target segmentation result of the image based on the associated information.

[0216] In one or more embodiments of this application, optionally, the number of target objects is at least one, and the processing module is further configured to:

[0217] Based on the target segmentation results of the video, target videos for each target object are obtained; wherein, the target video of the target object is the video obtained after removing the background of the target object from the video to be processed, or the target video is composed of multiple frames of images in the video to be processed containing the target object;

[0218] Output the target video and / or at least a portion of the images from the target video for each target object.

[0219] In one or more embodiments of this application, optionally, when the processing module outputs the target video and / or at least some images in the target video of each target object, it is specifically used to:

[0220] Display the target video and / or at least a portion of the images in the target video;

[0221] Based on the user's editing operations on the target video and / or at least some of the images in the target video, a promotional video and / or promotional images of the target object are generated.

[0222] In one or more embodiments of this application, optionally, the processing module's processing operations on the images in the video include:

[0223] The process involves acquiring the image to be processed from the video and the current loop features, and extracting the image features corresponding to the image; wherein the loop features are determined based on the processed images in the video.

[0224] Based on the image features and the loop features, the target segmentation result of the image to be processed is determined, and the image features and the loop features are input into the loop feature update model to obtain the updated loop features, so as to process the image to be processed in the next frame of the video according to the updated loop features.

[0225] In one or more embodiments of this application, optionally, the determining module is specifically used for:

[0226] The video to be processed is acquired, and at least one reference image in the video and the target segmentation result corresponding to each reference image are determined.

[0227] The method further includes: determining initial cyclic features based on the at least one frame of reference image and the corresponding target segmentation result;

[0228] The target segmentation result of the reference image is determined by the user annotating the target object in the reference image, or by automatically annotating the reference image based on a preset image of the target object.

[0229] The images to be processed in the video are images other than the reference image; the initial cyclic features are used to segment the first frame of the image to be processed.

[0230] In one or more embodiments of this application, optionally, the determining module is further configured to perform at least one of the following:

[0231] A reference image from the video is displayed so that the user can mark the location of the target object on the reference image, and the target segmentation result of the reference image is determined based on the user's markings;

[0232] Semantic segmentation is performed on the reference image in the video to determine at least one object in the reference image; the image regions corresponding to the at least one object are compared with preset images of target objects to obtain the target segmentation result of the reference image;

[0233] Semantic segmentation is performed on the reference image in the video to identify at least one object in the reference image; the image region corresponding to the at least one object is displayed, the target object selected by the user from the at least one object is obtained, and the target segmentation result is determined based on the image region corresponding to the target object.

[0234] In one or more embodiments of this application, optionally, when the processing module inputs the image features and cyclic features into the cyclic feature update model to obtain the updated cyclic features, it is specifically used for:

[0235] The image features and cyclic features are input into the extraction module to obtain the extraction result; wherein, the extraction module is used to fuse the image features and cyclic features;

[0236] The refining result is input into the enhancement module to obtain the enhancement result; wherein, the enhancement module is used to perform a pooling operation on the refining result;

[0237] The extraction result and the enhancement result are added together and then input into the compression module to obtain the updated loop feature of the preset length.

[0238] In one or more embodiments of this application, optionally, the extraction module includes a downsampling layer, a convolutional layer, a matrix calculation unit, and a normalization layer; the image features and cyclic features are input into the extraction module to obtain the extraction result, including:

[0239] The image features and cyclic features are downsampled by a downsampling layer, and the downsampled image features and cyclic features are respectively input into a convolutional layer to obtain a first feature matrix corresponding to the image features and a second feature matrix corresponding to the cyclic features.

[0240] The matrix computation unit performs matrix correlation operations on the first and second feature matrices, and inputs the results into the normalization layer to obtain the refined result. The video target segmentation apparatus provided in this application embodiment can be used to perform the above-described operation. Figures 1 to 7 The technical solutions of the embodiments shown are similar in principle and in effect, and will not be described again here.

[0241] This application embodiment also provides another video target segmentation apparatus, the apparatus may include:

[0242] The target product determination module is used to determine the video to be processed and the target product to be segmented in the video.

[0243] The product video processing module is used to process each frame of the video to be processed sequentially based on cyclic features using a pre-built target segmentation model, and to segment the target product from each frame of the video; wherein, the target segmentation model includes a cyclic feature update model, which is used to update the cyclic features according to the currently processed image during the video processing;

[0244] The generation module is used to generate target videos and / or target images of target products based on the target segmentation results of the video.

[0245] In one or more embodiments of this application, optionally, the target image is an image of the target object in the video to be processed, or an image obtained by removing the background from the image of the target object; the generation module is further configured to:

[0246] Obtain the filtering conditions input by the user, and filter the target image according to the filtering conditions;

[0247] The selected images are updated to a media library, which is used to create promotional videos and / or images for the target product. The video target segmentation apparatus provided in this application embodiment can be used to perform the above-described operation. Figure 8 The technical solutions of the embodiments shown are similar in principle and in effect, and will not be described again here.

[0248] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device in this embodiment may include:

[0249] At least one processor 901; and

[0250] Memory 902 communicatively connected to the at least one processor;

[0251] The memory 902 stores instructions that can be executed by the at least one processor 901 to cause the electronic device to perform the method as described in any of the above embodiments.

[0252] Alternatively, the memory 902 can be either standalone or integrated with the processor 901.

[0253] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0254] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the foregoing embodiments.

[0255] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0256] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0257] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0258] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0259] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0260] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0261] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0262] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0263] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0264] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A video target segmentation method, characterized in that, include: Determine the video to be processed and the target objects to be segmented in the video; By using a pre-built target segmentation model, each frame of the video to be processed is processed sequentially based on cyclic features to segment the target object from each frame; wherein, the target segmentation model includes a cyclic feature update model, which is used to update the cyclic features according to the currently processed image during video processing; The processing operations for the images in the video include: The process involves acquiring the image to be processed in the video and the current loop features, and extracting the image features corresponding to the image; wherein the loop features are features determined based on the processed images in the video. Based on the image features and cyclic features, the target segmentation result of the image to be processed is determined; The image features and loop features are then input into the loop feature update model to obtain the updated loop features, which are then used to process the next frame of the video image to be processed. The image features and cyclic features are input into the cyclic feature update model to obtain the updated cyclic features, including: The image features and cyclic features are input into the extraction module to obtain the extraction result; wherein, the extraction module is used to fuse the image features and cyclic features; The refining result is input into the enhancement module to obtain the enhancement result; wherein, the enhancement module is used to perform a pooling operation on the refining result; The extraction result and the enhancement result are added together and then input into the compression module to obtain the updated loop feature of the preset length.

2. The method according to claim 1, characterized in that, The target segmentation model further includes: an encoder, a feature reading module, and a decoder; Specifically, when processing any frame of an image in a video, the encoder is used to extract the image features corresponding to that image; The feature reading module is used to calculate the association information between the image features and the current loop features; The decoder is used to obtain the target segmentation result of the image based on the associated information.

3. The method according to claim 1, characterized in that, The number of target objects is at least one, and the method further includes: Based on the target segmentation results of the video, target videos for each target object are obtained; wherein, the target video of the target object is the video obtained after removing the background of the target object from the video to be processed, or the target video is composed of multiple frames of images in the video to be processed containing the target object; Output the target video and / or at least a portion of the images from the target video for each target object.

4. The method according to claim 3, characterized in that, Output the target video and / or at least a portion of the images from the target video for each target object, including: Display the target video and / or at least a portion of the images in the target video; Based on the user's editing operations on the target video and / or at least some of the images in the target video, a promotional video and / or promotional images of the target object are generated.

5. The method according to claim 1, characterized in that, Determine the video to be processed and the target objects to be segmented within the video, including: The video to be processed is acquired, and at least one reference image in the video and the target segmentation result corresponding to each reference image are determined. The method further includes: determining initial cyclic features based on the at least one frame of reference image and the corresponding target segmentation result; The target segmentation result of the reference image is determined by the user annotating the target object in the reference image, or by automatically annotating the reference image based on a preset image of the target object. The images to be processed in the video are images other than the reference image; the initial cyclic features are used to segment the first frame of the image to be processed.

6. The method according to claim 5, characterized in that, It also includes at least one of the following: A reference image from the video is displayed so that the user can mark the location of the target object on the reference image, and the target segmentation result of the reference image is determined based on the user's markings; Semantic segmentation is performed on the reference image in the video to determine at least one object in the reference image; the image regions corresponding to the at least one object are compared with preset images of target objects to obtain the target segmentation result of the reference image; Semantic segmentation is performed on the reference image in the video to identify at least one object in the reference image; the image region corresponding to the at least one object is displayed, the target object selected by the user from the at least one object is obtained, and the target segmentation result is determined based on the image region corresponding to the target object.

7. The method according to claim 1, characterized in that, The extraction module includes a downsampling layer, a convolutional layer, a matrix calculation unit, and a normalization layer; the image features and cyclic features are input into the extraction module to obtain the extraction results, including: The image features and cyclic features are downsampled by a downsampling layer, and the downsampled image features and cyclic features are respectively input into a convolutional layer to obtain a first feature matrix corresponding to the image features and a second feature matrix corresponding to the cyclic features. The matrix computation unit performs matrix correlation operations on the first and second feature matrices, and inputs the results into the normalization layer to obtain the refined results.

8. A video target segmentation method, characterized in that, include: Identify the video to be processed and the target products to be segmented within the video; By using a pre-built target segmentation model, each frame of the video to be processed is processed sequentially based on cyclic features to segment the target product from each frame of the video; wherein, the target segmentation model includes a cyclic feature update model, which is used to update the cyclic features according to the currently processed image during video processing; Based on the target segmentation results of the video, generate a target video and / or target image of the target product; The processing operations for the images in the video include: The process involves acquiring the image to be processed in the video and the current loop features, and extracting the image features corresponding to the image; wherein the loop features are features determined based on the processed images in the video. Based on the image features and cyclic features, the target segmentation result of the image to be processed is determined; The image features and loop features are then input into the loop feature update model to obtain the updated loop features, which are then used to process the next frame of the video image to be processed. The image features and cyclic features are input into the cyclic feature update model to obtain the updated cyclic features, including: The image features and cyclic features are input into the extraction module to obtain the extraction result; wherein, the extraction module is used to fuse the image features and cyclic features; The refining result is input into the enhancement module to obtain the enhancement result; wherein, the enhancement module is used to perform a pooling operation on the refining result; The extraction result and the enhancement result are added together and then input into the compression module to obtain the updated loop feature of the preset length.

9. The method according to claim 8, characterized in that, The target image is the image containing the target product in the video to be processed, or an image obtained by removing the background from the image containing the target product; the method further includes: Obtain the filtering conditions input by the user, and filter the target image according to the filtering conditions; The selected images are updated to the media library, which is used to create promotional videos and / or images for the target product.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the electronic device to perform the method according to any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-9.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

Images