Target segmentation and tracking method, material generation method and naked eye 3D scene card material

Abstract

The invention discloses a target segmentation and tracking method, a material generation method and a naked eye 3D scene card material, and relates to the technical field of information. The method comprises the following steps: acquiring a to-be-processed video frame sequence and a forward point added by an operator on a starting frame image of the video frame sequence for a target object; based on the forward point and a preset image segmentation model, performing target segmentation on the initial frame image to obtain a segmentation result of the target object; and according to the segmentation result and a preset target tracking model, performing continuous target tracking and segmentation on each video frame image in the video frame sequence to obtain a tracking result of the target object. The problems of high initial frame labeling cost and low segmentation precision of a VOS algorithm can be solved, meanwhile, the production efficiency of naked eye 3D scene card materials can be improved, and manpower resources and time cost are saved.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a target segmentation and tracking, material generation method and naked-eye 3D scene card material. Background Art

[0002] Object segmentation and tracking technology is the key technology for various video applications, such as intelligent monitoring, autonomous driving, virtual reality, etc. These application scenarios have increasingly higher requirements for object segmentation and tracking technology, requiring more accurate, efficient, and real-time solutions.

[0003] At present, the VOS (visual object tracking) algorithm is usually used to achieve the dual tasks of video segmentation and tracking. However, the VOS algorithm requires a fine initial frame annotation, which is often very expensive. In addition, the VOS algorithm is limited by the number of model parameters and its target segmentation accuracy is also poor. Summary of the invention

[0004] In view of this, the present application provides a target segmentation and tracking, material generation method and naked-eye 3D scene card material, the main purpose of which is to solve the problems of high initial frame annotation cost and low segmentation accuracy of the VOS algorithm.

[0005] According to a first aspect of the present application, a target segmentation and tracking method is provided, the method comprising:

[0006] Acquire a video frame sequence to be processed, and a positive point added by an operator for a target object on a starting frame image of the video frame sequence;

[0007] Based on the forward point and a preset image segmentation model, performing target segmentation on the starting frame image to obtain a segmentation result of the target object;

[0008] According to the segmentation result and the preset target tracking model, continuous target tracking and segmentation are performed on each video frame image in the video frame sequence to obtain the tracking result of the target object.

[0009] Optionally, performing target segmentation on the starting frame image based on the forward point and a preset image segmentation model to obtain a segmentation result of the target object includes:

[0010] Taking the forward point as a local key point of the target object in the starting frame image;

[0011] Based on the local key points, using the preset image segmentation model, performing target segmentation on the starting frame image to obtain a mask image corresponding to the starting frame image;

[0012] Based on the mask image corresponding to the starting frame image, a segmentation result of the target object is determined.

[0013] Optionally, the preset image segmentation model includes a local feature extraction sub-model and a target segmentation sub-model, and the target segmentation is performed on the starting frame image based on the local key points using the preset image segmentation model to obtain a mask image corresponding to the starting frame image, including:

[0014] According to the local key point, extracting local features around the local key point using the local feature extraction sub-model;

[0015] Based on the local features, using the target segmentation sub-model to perform target segmentation on the starting frame image to obtain a local area mask image;

[0016] The local area mask image is aggregated according to the position information of the local key points to obtain a mask image corresponding to the starting frame image.

[0017] Optionally, the performing continuous target tracking and segmentation on each video frame image in the video frame sequence according to the segmentation result and a preset target tracking model to obtain the tracking result of the target object includes:

[0018] Based on the mask image corresponding to the starting frame image, using the preset target tracking model, mask prediction is performed on the target object in each of the video frame images to obtain the mask image corresponding to each of the video frame images;

[0019] The tracking result of the target object is determined according to the mask image corresponding to each video frame image.

[0020] Optionally, obtaining a video frame sequence to be processed includes:

[0021] Get the video uploaded by the operator;

[0022] Analyzing comprehensive parameters of the video;

[0023] Based on the comprehensive parameters, determining whether the video meets the parameter requirements;

[0024] If the video meets the parameter requirements, the video frame sequence is determined according to the video.

[0025] Optionally, determining the video frame sequence according to the video includes:

[0026] Obtaining a start frame and an end frame selected by the operator for the video;

[0027] The video is intercepted based on the start frame and the end frame to obtain a video frame sequence.

[0028] Optionally, after performing target segmentation on the starting frame image based on the forward point and the preset image segmentation model to obtain the segmentation result of the target object, the method further includes:

[0029] If the segmentation result does not meet the preset segmentation standard, obtaining negative points added by the operator on the starting frame image for the target object;

[0030] Re-segmenting the target object on the starting frame image according to the positive point and the negative point to obtain a new segmentation result of the target object;

[0031] The step of continuously tracking and segmenting each video frame image in the video frame sequence according to the segmentation result and the preset target tracking model to obtain the tracking result of the target object includes:

[0032] According to the new segmentation result and the preset target tracking model, continuous target tracking and segmentation are performed on each video frame image in the video frame sequence to obtain the tracking result of the target object.

[0033] Optionally, the performing continuous target tracking and segmentation on each video frame image in the video frame sequence according to the new segmentation result and the preset target tracking model to obtain the tracking result of the target object includes:

[0034] In response to a new layer adding instruction triggered by the operator, determining a new layer according to the new segmentation result;

[0035] Based on the newly added layer and the preset target tracking model, continuous target tracking and segmentation are performed on each video frame image in the video frame sequence to obtain the tracking result of the target object.

[0036] According to a second aspect of the present application, a material generation method is provided, comprising:

[0037] Obtaining a segmentation result and a tracking result of a target object in a video frame sequence to be processed, wherein the segmentation result and the tracking result are determined by the target segmentation and tracking method described above;

[0038] Based on the segmentation result and the tracking result, generating a transparent channel video frame sequence corresponding to the video frame sequence;

[0039] Generate naked-eye 3D scene card material according to the transparent channel video frame sequence and the video frame sequence.

[0040] Optionally, the generating a transparent channel video frame sequence corresponding to the video frame sequence based on the segmentation result and the tracking result includes:

[0041] Binarizing the segmentation result and the tracking result to obtain a binary image corresponding to each video frame image in the video frame sequence;

[0042] Determine, according to the binarized image, the transparent channel video frame image corresponding to each of the video frame images;

[0043] The transparent channel video frame sequence is determined based on the transparent channel video frame images corresponding to the respective video frame images.

[0044] Optionally, generating a naked-eye 3D scene card material according to the transparent channel video frame sequence and the video frame sequence includes:

[0045] Each of the video frame images is spliced ​​left and right with the corresponding transparent channel video frame image to obtain the naked eye 3D scene card material.

[0046] According to a third aspect of the present application, a naked-eye 3D scene card material is provided, and the material is produced by the above-mentioned material generation method.

[0047] According to a fourth aspect of the present application, a target segmentation and tracking device is provided, comprising:

[0048] An acquisition unit, used for acquiring a video frame sequence to be processed, and a positive point added by an operator for a target object on a starting frame image of the video frame sequence;

[0049] A segmentation unit, configured to perform target segmentation on the starting frame image based on the forward point and a preset image segmentation model to obtain a segmentation result of the target object;

[0050] The tracking unit is used to perform continuous target tracking and segmentation on each video frame image in the video frame sequence according to the segmentation result and a preset target tracking model to obtain the tracking result of the target object.

[0051] Optionally, the segmentation unit includes:

[0052] A first determination submodule, configured to use the forward point as a local key point of the target object in the starting frame image;

[0053] A segmentation submodule, configured to perform target segmentation on the starting frame image based on the local key points and using the preset image segmentation model to obtain a mask image corresponding to the starting frame image;

[0054] The first determination submodule is further used to determine the segmentation result of the target object based on the mask image corresponding to the starting frame image.

[0055] Optionally, the preset image segmentation model includes a local feature extraction sub-model and a target segmentation sub-model.

[0056] The segmentation submodule is specifically used to extract local features around the local key points based on the local key points using the local feature extraction submodel; based on the local features, use the target segmentation submodel to perform target segmentation on the starting frame image to obtain a local area mask image; according to the position information of the local key points, aggregate the local area mask images to obtain a mask image corresponding to the starting frame image.

[0057] Optionally, the tracking unit is specifically used to perform mask prediction on the target object in each video frame image based on the mask image corresponding to the starting frame image and using the preset target tracking model to obtain the mask image corresponding to each video frame image; and determine the tracking result of the target object according to the mask image corresponding to each video frame image.

[0058] Optionally, the acquisition unit includes:

[0059] The acquisition submodule is used to obtain the video uploaded by the operator;

[0060] An analysis submodule, used for analyzing comprehensive parameters of the video;

[0061] A determination submodule, used to determine whether the video meets the parameter requirements based on the comprehensive parameters;

[0062] The second determination submodule is used to determine the video frame sequence according to the video if the video meets the parameter requirements.

[0063] Optionally, the second determination submodule is specifically used to obtain a start frame and an end frame selected by the operator for the video; and to intercept the video based on the start frame and the end frame to obtain a video frame sequence.

[0064] Optionally, the acquisition unit is further configured to acquire negative points added by the operator on the starting frame image for the target object if the segmentation result does not meet a preset segmentation standard;

[0065] The segmentation unit is further used to re-segment the target object on the starting frame image according to the positive point and the negative point to obtain a new segmentation result of the target object;

[0066] The tracking unit is specifically used to perform continuous target tracking and segmentation on each video frame image in the video frame sequence according to the new segmentation result and the preset target tracking model to obtain the tracking result of the target object.

[0067] Optionally, the tracking unit is further specifically used to respond to a new layer instruction triggered by the operator, determine a new layer according to the new segmentation result; based on the new layer and a preset target tracking model, perform continuous target tracking and segmentation on each video frame image in the video frame sequence to obtain the tracking result of the target object.

[0068] According to a fifth aspect of the present application, a material generating device is provided, comprising:

[0069] An acquisition unit, used for acquiring a segmentation result and a tracking result of a target object in a video frame sequence to be processed, wherein the segmentation result and the tracking result are determined by the target segmentation and tracking method described above;

[0070] A first generating unit, configured to generate a transparent channel video frame sequence corresponding to the video frame sequence based on the segmentation result and the tracking result;

[0071] The second generating unit is used to generate naked-eye 3D scene card materials according to the transparent channel video frame sequence and the video frame sequence.

[0072] Optionally, the first generating unit is specifically used to binarize the segmentation result and the tracking result to obtain a binarized image corresponding to each video frame image in the video frame sequence; determine the transparent channel video frame image corresponding to each video frame image based on the binarized image; and determine the transparent channel video frame sequence based on the transparent channel video frame images corresponding to each video frame image.

[0073] Optionally, the second generating unit is specifically configured to respectively splice each of the video frame images with the corresponding transparent channel video frame images on the left and right sides to obtain the naked-eye 3D scene card material.

[0074] According to a sixth aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned target segmentation and tracking method is implemented.

[0075] According to the seventh aspect of the present application, a computer device is provided, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned target segmentation and tracking method when executing the computer program.

[0076] According to an eighth aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned material generation method is implemented.

[0077] According to the ninth aspect of the present application, a computer device is provided, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned material generation method when executing the computer program.

[0078] By means of the above technical solution, the target segmentation and tracking, material generation method and naked-eye 3D scene card material provided by the present application, compared with the prior art, can realize the segmentation and tracking of target objects in the video by combining a preset image segmentation model and a preset target tracking model. Since the preset image segmentation model of the present application can complete the accurate segmentation of the target object based only on the forward points added by the operator, and does not require fine annotation of the initial frame, it can solve the problem of high initial frame annotation cost of the VOS algorithm and reduce the target segmentation and tracking cost of the video. At the same time, since the present application uses a preset image segmentation model alone for target segmentation, the number of model parameters is far greater than that of the existing VOS algorithm model, so the precision of the target edge segmentation can be guaranteed, thereby overcoming the problem of low segmentation accuracy of the VOS algorithm and improving the segmentation accuracy of the target object in the video.

[0079] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0081] Figure 1 A schematic diagram of a flow chart of a target segmentation and tracking method provided in an embodiment of the present application is shown;

[0082] Figure 2 A schematic diagram of video uploading provided in an embodiment of the present application is shown;

[0083] Figure 3 A schematic diagram of the forward point addition and segmentation effect provided by an embodiment of the present application is shown;

[0084] Figure 4 A schematic diagram of the segmentation effect of the SAM model provided in an embodiment of the present application is shown;

[0085] Figure 5 A schematic diagram of the segmentation effect of the SiamMask model provided in an embodiment of the present application is shown;

[0086] Figure 6 A schematic diagram of target tracking results provided by an embodiment of the present application is shown;

[0087] Figure 7 A schematic diagram of video parameters provided in an embodiment of the present application is shown;

[0088] Figure 8 A schematic diagram of a start frame and an end frame provided in an embodiment of the present application is shown;

[0089] Fig. 9 A schematic diagram of the out-of-frame effect provided by an embodiment of the present application is shown;

[0090] Fig.10 A schematic diagram of the technical process of presenting a 3D visual effect provided by an embodiment of the present application is shown;

[0091] Fig.11 A schematic diagram of a forward link provided by an embodiment of the present application is shown;

[0092] Fig.12 A schematic diagram of a reverse link provided by an embodiment of the present application is shown;

[0093] Fig.13 A schematic diagram of a material generation method provided in an embodiment of the present application is shown;

[0094] Fig.14 A schematic diagram of a naked-eye 3D scene card material provided in an embodiment of the present application is shown;

[0095] Fig.15 A schematic diagram of the structure of a target segmentation and tracking device provided in an embodiment of the present application is shown;

[0096] Fig.16 A schematic diagram of the physical structure of a computer device provided by an embodiment of the present invention is shown;

[0097] Fig.17 A schematic structural diagram of a material generating device provided in an embodiment of the present application is shown;

[0098] Fig.18 A schematic diagram of the physical structure of another computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0099] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0100] In order to solve the problem of high initial frame annotation cost and low segmentation accuracy of the VOS algorithm. This embodiment provides a target segmentation and tracking method, such as Figure 1 As shown, the method includes:

[0101] Step 101: Obtain a video frame sequence to be processed and a positive point added by an operator for a target object on a starting frame image of the video frame sequence.

[0102] Among them, the video frame sequence is a video frame sequence uploaded by the operator for target segmentation and tracking. The video frame sequence can specifically come from advertising videos, surveillance videos and videos in other fields. The video frame sequence includes each video frame image from the start frame to the end frame. The forward point is used to select the target object to be segmented and tracked. The number of target objects can be one, two or more. The target objects include diversified targets such as dolls, people, vehicles, flowers, buildings, etc. It should be noted that the target objects in the embodiments of the present invention are not limited to the dolls, people, vehicles, flowers, buildings listed above, but can also be other objects, and the embodiments of the present invention do not specifically limit this.

[0103] For the embodiment of the present invention, the operator can upload the video to be processed to the video processing system, such as Figure 2 After receiving the video, the video processing system will perform frame processing on the video to obtain a video frame sequence, and display the starting frame image in the video frame sequence to the operator. The operator can add positive points (Positive) in the starting frame image by clicking, and can select the target object to be segmented and tracked, such as Figure 3 As shown, the point on the gift box is the positive point, and the gift box is the selected target object.

[0104] Step 102: Based on the forward points and a preset image segmentation model, target segmentation is performed on the starting frame image to obtain a segmentation result of the target object.

[0105] The preset image segmentation model may specifically be SAM (Segment Anything Model) or other segmentation models, which is not specifically limited in the embodiment of the present invention.

[0106] For the embodiment of the present invention, after obtaining the positive points added by the operator for the target object, the starting frame image is segmented based on the positive points using the preset image segmentation model to obtain the segmentation result of the target object. For this process, step 102 specifically includes: using the positive points as local key points of the target object in the starting frame image; based on the local key points, using the preset image segmentation model, the starting frame image is segmented to obtain the mask image corresponding to the starting frame image; based on the mask image corresponding to the starting frame image, the segmentation result of the target object is determined.

[0107] The preset image segmentation model includes a local feature extraction sub-model and a target segmentation sub-model, the local feature extraction sub-model may be a prompt encoder, and the target segmentation sub-model may be a CNN model or a Transformer model, etc. It should be noted that the local feature extraction sub-model and the target segmentation sub-model in the embodiment of the present invention are not limited to the above-mentioned models, and may also be other local feature extraction sub-models and target segmentation sub-models.

[0108] When performing target segmentation on the starting frame image based on local key points using a preset image segmentation model, local features around the local key points are extracted using the local feature extraction submodel according to the local key points; based on the local features, the target segmentation submodel is used to perform target segmentation on the starting frame image to obtain a local area mask image; based on the position information of the local key points, the local area mask images are aggregated to obtain a mask image corresponding to the starting frame image.

[0109] Now, we will take the SAM model as an example to illustrate the segmentation process of the target object. First, we will initialize the SAM model and obtain the positive points added by the operator on the system to determine the target object to be segmented. Since the SAM model adopts a local feature aggregation strategy, the obtained positive points can be used as local key points of the target object in the starting frame image. The number of positive points can be one, two or more. Then, the local feature extraction submodel in the SAM model is used to extract the local features around the local key points in the starting frame image. Then, based on the local features, the target segmentation submodel in the SAM model is used to perform target segmentation on the starting frame image to obtain a local mask image. Finally, based on the position of the local key points, all local area mask images are aggregated to obtain the mask image corresponding to the starting frame image, so that the segmentation result of the target object can be determined. Figure 3 Taking the segmentation result in as an example, the gift box is the target object, and the points on the gift box are the positive points selected and added by the operator. Based on the positive points, SAM is used to segment the gift box in the starting frame image, and finally the mask image of the gift box can be obtained.

[0110] Since the SAM model adopts an effective local feature aggregation strategy, it can segment the target with higher quality and finer resolution. Through the added positive points, SAM can adaptively determine the scale and segmentation degree and obtain a highly accurate target mask.

[0111] The SAM model of the embodiment of the present invention represents the latest progress in the field of AI. It adopts the "big model, big data" strategy and trains multiple versions based on the ViT architecture, including ViT-base, ViT-large and ViT-huge. Compared with the traditional CNN model, the ViT architecture stands out for its advantages in depth expansion and flexibility of model parameters. Taking the SiamMask (VideoObject Tracking / Segmentation) model as an example, although it is based on the classic ResNet-18 convolutional structure, its parameters are limited, while the largest version of the SAM model (ViT-huge architecture) has more than 600 million parameters, which is an order of magnitude higher than the SiamMask model. The SAM model in the embodiment of the present invention fully demonstrates the key role of the "big model" in achieving excellent performance. At the same time, in the embodiment of the present invention, the SAM model is combined with AI for data annotation, which realizes the generation of 1 billion mask images (masks) from 11 million original images and constructs a large-scale data set. The scale of this dataset is 400 times that of the existing SiamMask segmentation dataset, providing sufficient learning resources for the SAM model, enabling it to fully utilize its large model's "appetite" to digest and absorb big data, thereby significantly improving the segmentation effect.

[0112] like Figure 4 and Figure 5 The segmentation effects of the SAM model and the SiamMask model are shown. When the SAM model is used to segment the target object, the contours of the doll with rich details and other objects of different shapes can be accurately captured, which can show the excellent accuracy and delicate processing ability of the SAM model in the segmentation task. However, the SiamMask model's processing of the doll's contour is relatively vague, which exposes its shortcomings in defining the target boundary.

[0113] Step 103: According to the segmentation result and a preset target tracking model, continuous target tracking and segmentation are performed on each video frame image in the video frame sequence to obtain a tracking result of the target object.

[0114] Among them, each video frame image includes all video frame images from the start frame to the end frame, and the preset target tracking model actually refers to the VOT (Visual Object Tracking) model, which specifically includes the Xmem model. The focus of the VOT model is to analyze the state of the object in the video sequence, and to stably locate the target when the object is disturbed by similar interferences, rapid motion, deformation, occlusion, etc. In short, the focus of the VOT model is the association between the previous and next frames of the target object in the sequence. It should be noted that the preset target tracking model is not limited to the Xmem model, and can also be other target tracking models, which is not specifically limited in the embodiments of the present invention.

[0115] For the embodiment of the present invention, although the preset image segmentation model (such as the SAM model) is good at target segmentation of a single frame image, it does not have the ability to track videos and cannot process videos. Based on this, the embodiment of the present invention adopts a strategy that combines a preset image segmentation model with a preset target tracking model. When using the preset target tracking model to track a target in a video frame sequence, step 103 specifically includes: based on the mask image corresponding to the starting frame image, using the preset target tracking model to perform mask prediction on the target object in each video frame image to obtain the mask image corresponding to each video frame image; according to the mask image corresponding to each video frame image, determine the tracking result of the target object.

[0116] like Figure 6 As shown in the figure, taking the combination of the SAM model and the VOT model as an example, after obtaining the segmentation result of the SAM model, the segmentation result of the SAM model is used as the input of the VOT model, that is, the VOT model is used to predict the mask of the target object in each video frame image in the video frame sequence. In normal scenarios, the system can output satisfactory results. When the quality of subsequent mask images does not meet the requirements, the intermediate parameters processed by the VOT model can be provided to the SAM model for refinement. Specifically, the correlation degree of the target object between each frame is provided to the SAM model as a hint. Based on this, the SAM model can generate an optimized mask image (Mask) to improve the processing quality of subsequent frames.

[0117] In some embodiments, based on the characteristics of the SAM model supporting a series of different interactive segmentation methods, the embodiments of the present invention use Python's Gradio to implement visual interactive operations, and combine vipserver and Tpp python to build a video processing system for target segmentation and tracking. When an operator uploads a video, the video system can instantly analyze and feedback the comprehensive parameters of the video, such as the total number of frames, frame rate (Frames Per Second, FPS), picture size and other basic information, to provide a basis for subsequent refined processing. Based on this, the method includes: obtaining the video uploaded by the operator; analyzing the comprehensive parameters of the video; based on the comprehensive parameters, determining whether the video meets the parameter requirements; if the video meets the parameter requirements, then determining the video frame sequence according to the video. Among them, the comprehensive parameters include: video name, total number of frames, frame rate, picture size and other basic information.

[0118] like Figure 7 As shown, when the total number of frames, or the frame rate, or the resolution size exceeds a certain value, it means that the video does not meet the parameter requirements. At this time, the operator can be prompted to wait for a while if the processing is slow due to the limited performance of the installed server. The embodiment of the present invention can screen out videos that meet the requirements by analyzing the comprehensive parameters of the uploaded video.

[0119] In some embodiments, the start frame and the end frame can be freely set according to actual needs. Based on this, the method further includes: obtaining the start frame and the end frame selected by the operator for the video; intercepting the video based on the start frame and the end frame to obtain a video frame sequence. Figure 8 As shown, the starting frame is 201 and the ending frame is 350. Based on the starting frame and the ending frame, a video segment can be captured from the uploaded video frame to obtain a video frame sequence.

[0120] In an embodiment of the present invention, the operator can not only freely select the target object in the video according to actual service needs, but also freely set the starting frame and ending frame of the video processing. In this way, even if the target object does not appear at the beginning of the video for the first time, any period of its appearance can be accurately captured, thereby ensuring that the processing of each frame can focus on the core communication information.

[0121] In some embodiments, the video processing system allows the user to specify the target objects that need to be retained or emphasized in the video, such as brand logos, specific product models, etc., to ensure that these elements can still be clearly presented after processing. It also allows users to clearly mark non-target objects or interference factors that they want to remove from the processing results, such as irrelevant people in the background, uncoordinated color spots, etc. Based on this, the method also includes: if the segmentation result does not meet the preset segmentation standard, then obtain the negative points added by the operator on the starting frame image for the target object; according to the positive points and the negative points, re-segment the starting frame image to obtain a new segmentation result of the target object; at the same time, according to the new segmentation result and the preset target tracking model, each video frame image in the video frame sequence is continuously tracked and segmented to obtain the tracking result of the target object.

[0122] by Figure 8 Take the flower gift box in the video as an example. The operator wants to select the flower gift box as the target object to be segmented. Therefore, a positive point Positive is added to the flower gift box. After segmentation using the preset image segmentation model, if the segmentation result includes the flower gift box as well as the surrounding petals or other objects, it means that the segmentation effect does not meet the preset segmentation standard. At this time, the operator can add a negative point Negative to eliminate the interference of the surrounding petals or other objects. After obtaining the positive and negative points added by the operator, the system will reuse the preset image segmentation model for target segmentation to achieve the expected segmentation effect. Then, based on the new segmentation result, the preset target tracking model is used to continuously track and segment the video frame sequence to obtain the tracking result of the target object.

[0123] By combining two complementary interaction modes, the video processing system of the embodiment of the present invention not only greatly optimizes the processing accuracy, but also can effectively reduce the negative impact of unexpected elements on video processing by intelligently identifying and separating target content and non-target content, thereby ensuring the purity and professionalism of visual communication. This two-way interactive selection mechanism not only reflects the flexibility of the system, but also highlights its excellent ability to meet the personalized needs of users.

[0124] In some embodiments, if the segmentation result meets expectations, the preset target tracking model can be enabled to continuously track the target object in the video frame sequence by adding a new layer. For this process, the method includes: responding to the new layer instruction triggered by the operator, determining the new layer according to the new segmentation result; based on the new layer and the preset target tracking model, continuously tracking and segmenting each video frame image in the video frame sequence to obtain the tracking result of the target object.

[0125] like Figure 7 As shown, after the preset image segmentation model segments the target object based on positive points and negative points, if the segmentation result meets expectations, the operator can click the button to add a new layer. At this time, the system will receive the instruction to add a new layer, and based on the instruction to add a new layer, the segmented mask image will be used as a new layer and passed to the preset target tracking model for further tracking. If the operator feels that the new layer selection is wrong, the system can click the Clear Selected Layer button. After receiving the instruction to clear the selected layer, the system will clear the new layer based on the instruction to clear the selected layer for the operator to reselect. In addition, the operator can clear the added positive points and negative points by clicking the Clear Click button.

[0126] It should be noted that if there are at least two target objects in the video sequence, it is necessary to add forward points and increase layers respectively. After all layers are added, the preset target tracking model is used to perform multi-target tracking.

[0127] The present embodiment provides a target segmentation and tracking method, which can achieve the segmentation and tracking of target objects in the video by combining a preset image segmentation model and a preset target tracking model. Since the preset image segmentation model of the embodiment of the present invention can complete the accurate segmentation of the target object based only on the forward points added by the operator, and does not require fine annotation of the initial frame, it can solve the problem of high initial frame annotation cost of the VOS algorithm and reduce the target segmentation and tracking cost of the video. At the same time, since the present application solely uses a preset image segmentation model for target segmentation, the number of model parameters is far greater than that of the existing VOS algorithm model, and therefore the precision of the target edge segmentation can be guaranteed, thereby overcoming the problem of low segmentation accuracy of the VOS algorithm and improving the segmentation accuracy of the target object in the video.

[0128] Furthermore, in this era where innovation and technology converge, the advertising field is undergoing a profound transformation, breaking through the traditional two-dimensional limitations. The naked-eye 3D effect, with its stunning visual effects, redefines the form of advertising. Advertising is no longer just a carrier of information, it has evolved into a thrilling sensory feast, inviting every viewer to immerse themselves in it. The naked-eye 3D effect, as the name suggests, means that there is no need for additional equipment, and the three-dimensional image can be felt with the naked eye alone. Unlike traditional 3D technology that requires wearing special glasses, the naked-eye 3D effect uses the principles of physics to directly present depth and layering on the screen, creating an immersive visual experience. The key to achieving the above-mentioned naked-eye 3D effect is to create the feeling that the elements in the picture break through the boundaries of the screen, the so-called "out-of-frame effect", such as Fig. 9 However, it is not easy to present the naked eye 3D visual effect well. There is a complex and precise technical chain behind it. This process can be roughly divided into three core steps: material creation, placement restrictions and terminal display. Fig.10 As shown. In the material creation stage, the goal is to batch generate content that meets the out-of-frame video standards while ensuring that the advertising cost will not increase. In order to overcome the above-mentioned material creation difficulties, the present invention provides a material generation method. The material generation methods are mainly divided into two categories: positive link and negative link. Among them, the positive link is to directly synthesize the video through the existing materials and templates without complicated cutouts, such as Fig.11 As shown, however, the materials used for APP advertisements are mostly 2D planes, and the videos generated by the forward link are insufficient in terms of 3D immersion, making it difficult for the audience to pay attention for a long time. For this reason, the material generation method provided by the embodiment of the present invention is mainly based on the reverse link, that is, the production of "out-of-frame videos" for "ready-made videos". The core idea is to first segment and track the target object, and then binarize the mask image of the target object to form a transparent channel video frame image, that is, an Alpha channel video frame image, and finally splice it into a "left RGB right Alpha channel" video, thereby generating naked-eye 3D scene card materials, such as Fig.12 shown.

[0129] The key to the reverse link is to accurately locate, segment and track the target object in the video, which requires the support of computer vision technology, especially the system functions of the three major tasks of target detection, tracking and segmentation. Based on this, the embodiment of the present invention proposes a material generation method based on the above target segmentation and tracking method. The specific process is as follows: Fig.13 As shown, including:

[0130] Step 201: Obtain the segmentation result and tracking result of the target object in the video frame sequence to be processed.

[0131] The segmentation result and the tracking result are determined by the target segmentation and tracking method. The video frame sequence may be from an advertising video, which may be an APP advertising video or other advertising videos, and the embodiment of the present invention does not specifically limit this.

[0132] Based on the characteristics that the preset image segmentation model (such as the SAM model) in Example 1 supports a series of different interactive segmentation methods, the embodiment of the present invention uses Python's Gradio to implement visual interactive operations, and combines vipserver with Tpp python to build a material generation system for naked-eye 3D scene cards that can be operated. The material generation system includes all the functions of the video processing system in Example 1.

[0133] When an operator uploads a video (such as an advertising video) in the material generation system, Figure 2As shown, the material generation system can analyze the comprehensive parameters of the video, including basic information such as video name, total number of frames, frame rate, and image size, and then filter out videos that meet the size and duration requirements of the homepage scene card based on the comprehensive parameters, such as Figure 7 shown.

[0134] At the same time, the operator can select the target object to be segmented and tracked on the initial frame image by adding positive points Positive, and can also remove non-target objects or interference factors on the initial frame image by adding negative points Negative. After obtaining the positive points and / or negative points, the material generation system can use the target segmentation method in Example 1, that is, use the preset image segmentation model to perform target segmentation, and obtain a mask image corresponding to the initial frame image, such as Figure 3 If the segmentation result reaches the expected effect, the operator can click the add layer button. At this time, the material generation system will receive the add layer instruction. Based on the add layer instruction, the mask graphic is used as the input of the preset target tracking model (such as the VOT model) in the first embodiment. The preset target tracking model will continue to track and segment the target. Figure 6 shown.

[0135] It should be noted that if there are at least two target objects, the operator needs to add positive points for each target object and add layers. When all layers are added, the preset target tracking model can perform multi-target tracking, thereby achieving simultaneous segmentation of multiple focus elements in the advertising video.

[0136] The material generation system of the embodiment of the present invention supports a two-way selection interactive design, which allows users to specify the target objects that need to be retained or emphasized in the video, such as brand logos, specific product models, etc., to ensure that these elements can still be clearly presented after processing, maintain the core communication intention of the advertisement, and enhance the delivery effect of the advertising information. Users can also clearly mark non-target objects or interference factors that they hope to propose from the processing results, such as irrelevant people in the background, uncoordinated color spots, etc. By combining these two complementary interaction modes, the material generation system of the embodiment of the present invention not only greatly optimizes the processing accuracy, but also effectively reduces the negative impact of unexpected elements on the quality of the finished product through intelligent identification and separation of target content and non-target content, ensuring the purity and professionalism of visual communication. This two-way interactive mechanism of the embodiment of the present invention not only reflects the flexibility of the system, but also highlights its excellent ability to meet the personalized needs of users and highlight the creative effects of advertisements.

[0137] In order to further improve the flexibility and pertinence of material production, the embodiments of the present invention can not only freely select the target object in the video according to the actual needs of different brand advertisements, but also freely set the start frame and end frame of the video processing, such as Figure 8 This design greatly reduces the creative design restrictions on material production. Even if the target object does not appear at the beginning of the video for the first time, it can accurately capture any time when it appears, ensuring that the processing of each frame can focus on the core communication information. This is especially important for advertising products that dynamically display product features, which can ensure that key visual moments are quickly located and strengthened in fast-paced narratives, improving the audience's viewing experience and information reception efficiency.

[0138] In summary, the embodiments of the present invention have brought revolutionary changes to the field of advertising production by integrating detailed video parameter preview, two-way selection interactive mode, multi-target tracking and free video selection functions. It simplifies the tedious manual material editing process, improves the material generation efficiency, and makes the production of creative content more efficient and accurate.

[0139] Step 202: Generate a transparent channel video frame sequence corresponding to the video frame sequence based on the segmentation result and the tracking result.

[0140] The segmentation result includes a mask image corresponding to the start frame image, and the tracking result includes a mask image corresponding to each video frame image in the video frame sequence. The transparent channel video frame sequence is specifically an Alpha channel video frame sequence.

[0141] For the embodiment of the present invention, in order to generate a transparent channel video frame sequence, step 202 specifically includes: binarizing the segmentation result and the tracking result to obtain a binarized image corresponding to each video frame image in the video frame sequence; determining the transparent channel video frame image corresponding to each video frame image based on the binarized image; and determining the transparent channel video frame sequence based on the transparent channel video frame images corresponding to each video frame image.

[0142] Specifically, each video frame image in the video frame sequence can be binarized. For example, the pixel value of the mask area corresponding to the target object is set to 1, and the pixel value of other areas is set to 0, thereby obtaining a binarized image corresponding to each video frame image, and then determining the binarized image as a transparent channel video frame image, thereby obtaining a transparent channel video frame sequence, such as Fig.14 shown.

[0143] Step 203: Generate naked-eye 3D scene card material according to the transparent channel video frame sequence and the video frame sequence.

[0144] For the embodiment of the present invention, after obtaining the transparent channel video frame sequence, each video frame image in the original video frame sequence and each video frame image in the transparent channel video frame sequence are spliced ​​left and right to obtain the naked eye 3D scene card material, such as Fig.14 shown.

[0145] A material generation method provided by an embodiment of the present invention generates a transparent channel video frame sequence corresponding to the original video frame sequence through the segmentation and tracking results of the target object, and splices the two to generate naked-eye 3D scene card materials, thereby improving the production efficiency of naked-eye 3D scene card materials and saving human resources and time costs.

[0146] Furthermore, the embodiment of the present invention also provides a naked-eye 3D scene card material, which can be produced by the material generation method in the second embodiment.

[0147] Further, as Figure 1 The specific implementation of the method shown in the embodiment provides a target segmentation and tracking device, such as Fig.15 As shown, the device includes: an acquisition unit 31, a segmentation unit 32 and a tracking unit 33.

[0148] The acquisition unit 31 may be used to acquire the video frame sequence to be processed and the forward points added by the operator for the target object on the starting frame image of the video frame sequence.

[0149] The segmentation unit 32 may be configured to perform target segmentation on the start frame image based on the forward points and a preset image segmentation model to obtain a segmentation result of the target object.

[0150] The tracking unit 33 may be configured to perform continuous target tracking and segmentation on each video frame image in the video frame sequence according to the segmentation result and a preset target tracking model, so as to obtain the tracking result of the target object.

[0151] In some embodiments, the segmentation unit 32 includes: a first determination submodule and a segmentation submodule.

[0152] The first determination submodule may be configured to use the forward point as a local key point of the target object in the starting frame image.

[0153] The segmentation submodule can be used to perform target segmentation on the starting frame image based on the local key points and using the preset image segmentation model to obtain a mask image corresponding to the starting frame image.

[0154] The first determination submodule may also be used to determine the segmentation result of the target object based on the mask image corresponding to the starting frame image.

[0155] In some embodiments, the preset image segmentation model includes a local feature extraction sub-model and a target segmentation sub-model.

[0156] The segmentation submodule can be specifically used to extract local features around the local key points based on the local key points using the local feature extraction submodel; based on the local features, use the target segmentation submodel to perform target segmentation on the starting frame image to obtain a local area mask image; based on the position information of the local key points, aggregate the local area mask images to obtain a mask image corresponding to the starting frame image.

[0157] In some embodiments, the tracking unit 33 can be specifically used to perform mask prediction on the target object in each video frame image based on the mask image corresponding to the starting frame image and using the preset target tracking model to obtain the mask image corresponding to each video frame image; and determine the tracking result of the target object according to the mask image corresponding to each video frame image.

[0158] In some embodiments, the acquisition unit 31 includes: an acquisition submodule, an analysis submodule, a determination submodule and a second determination submodule.

[0159] The acquisition submodule can be used to acquire the video uploaded by the operator.

[0160] The analysis submodule can be used to analyze the comprehensive parameters of the video.

[0161] The determination submodule may also be used to determine whether the video meets parameter requirements based on the comprehensive parameters.

[0162] The second determination submodule may be configured to determine the video frame sequence according to the video if the video meets the parameter requirement.

[0163] In some embodiments, the second determination submodule may be specifically used to obtain a start frame and an end frame selected by the operator for the video; and to capture the video based on the start frame and the end frame to obtain a video frame sequence.

[0164] In some embodiments, the acquisition unit 31 may also be configured to acquire negative points added by the operator on the starting frame image for the target object if the segmentation result does not meet a preset segmentation standard.

[0165] The segmentation unit 32 may also be used to re-segment the target object on the starting frame image according to the positive points and the negative points to obtain a new segmentation result of the target object.

[0166] The tracking unit 33 may be specifically configured to perform continuous target tracking and segmentation on each video frame image in the video frame sequence according to the new segmentation result and the preset target tracking model, so as to obtain the tracking result of the target object.

[0167] In some embodiments, the tracking unit 33 can also be specifically used to respond to a new layer instruction triggered by the operator, determine a new layer according to the new segmentation result; based on the new layer and the preset target tracking model, perform continuous target tracking and segmentation on each video frame image in the video frame sequence to obtain the tracking result of the target object.

[0168] It should be noted that for other corresponding descriptions of the functional units involved in the target segmentation and tracking device provided in this embodiment, reference can be made to Figure 1 The corresponding description in will not be repeated here.

[0169] Based on the above Figure 1 The method shown in the embodiment of the present application is accordingly provided with a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned Figure 1 The target segmentation and tracking method shown.

[0170] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0171] Based on the above Figure 1 The method shown, and Fig.15 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides a computer device, such as Fig.16 As shown, it can be a tablet computer, a smart phone, a smart watch, a smart bracelet, or other network devices, etc. The computer device includes a storage medium and a processor; the storage medium is used to store computer programs; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 The target segmentation and tracking method shown.

[0172] Optionally, the above-mentioned physical devices may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0173] Those skilled in the art will appreciate that the client device structure provided in this embodiment does not constitute a limitation on such physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0174] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the two physical devices mentioned above, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the information processing physical device.

[0175] The embodiment of the present invention can achieve the segmentation and tracking of target objects in the video by combining a preset image segmentation model and a preset target tracking model. Since the preset image segmentation model of the embodiment of the present invention can complete the accurate segmentation of the target object based only on the forward points added by the operator, and does not require fine annotation of the initial frame, it can solve the problem of high initial frame annotation cost of the VOS algorithm and reduce the target segmentation and tracking cost of the video. At the same time, since the present application solely uses a preset image segmentation model for target segmentation, the amount of its model parameters is far more than that of the existing VOS algorithm model, so it can ensure the fineness of the target edge segmentation, thereby overcoming the problem of low segmentation accuracy of the VOS algorithm and improving the segmentation accuracy of the target object in the video.

[0176] Further, as Fig.13 The specific implementation of the method shown in this embodiment provides a material generation device, such as Fig.17 As shown, the device includes: an acquisition unit 51, a first generation unit 52 and a second generation unit 53.

[0177] The acquisition unit 51 can be used to acquire the segmentation result and the tracking result of the target object in the video frame sequence to be processed, wherein the segmentation result and the tracking result are determined by the target segmentation and tracking method according to claim 1.

[0178] The first generating unit 52 may be configured to generate a transparent channel video frame sequence corresponding to the video frame sequence based on the segmentation result and the tracking result.

[0179] The second generating unit 53 may be configured to generate a naked-eye 3D scene card material according to the transparent channel video frame sequence and the video frame sequence.

[0180] In some embodiments, the first generating unit 52 can be specifically used to perform binarization processing on the segmentation result and the tracking result to obtain a binarized image corresponding to each video frame image in the video frame sequence; determine the transparent channel video frame image corresponding to each video frame image based on the binarized image; and determine the transparent channel video frame sequence based on the transparent channel video frame images corresponding to each video frame image.

[0181] In some embodiments, the second generating unit 53 may be specifically configured to respectively splice each of the video frame images with its corresponding transparent channel video frame image to obtain the naked eye 3D scene card material.

[0182] It should be noted that for other corresponding descriptions of the functional units involved in the material generation device provided in this embodiment, reference can be made to Fig.13 The corresponding description in will not be repeated here.

[0183] Based on the above Fig.13 The method shown in the embodiment of the present application is accordingly provided with a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned Fig.13 The material generation method shown.

[0184] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0185] Based on the above Fig.13 The method shown, and Fig.17 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides a computer device, such as Fig.18 As shown, it can be a tablet computer, a smart phone, a smart watch, a smart bracelet, or other network devices, etc. The computer device includes a storage medium and a processor; the storage medium is used to store computer programs; the processor is used to execute the computer program to achieve the above-mentioned Fig.13 The material generation method shown.

[0186] Optionally, the above-mentioned physical devices may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0187] Those skilled in the art will appreciate that the client device structure provided in this embodiment does not constitute a limitation on such physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0188] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the two physical devices mentioned above, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the information processing physical device.

[0189] The embodiment of the present invention generates a transparent channel video frame sequence corresponding to the original video frame sequence through the segmentation and tracking results of the target object, and splices the two to generate naked-eye 3D scene card materials. Naked-eye 3D scene card materials can be generated in batches without increasing advertising costs.

[0190] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0191] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A target segmentation and tracking method, characterized in that: include: Acquire a video frame sequence to be processed, and a positive point added by an operator for a target object on a starting frame image of the video frame sequence; Based on the forward point and a preset image segmentation model, performing target segmentation on the starting frame image to obtain a segmentation result of the target object; According to the segmentation result and the preset target tracking model, continuous target tracking and segmentation are performed on each video frame image in the video frame sequence to obtain the tracking result of the target object.

2. The method according to claim 1, characterized in that The performing target segmentation on the starting frame image based on the forward point and the preset image segmentation model to obtain the segmentation result of the target object includes: Taking the forward point as a local key point of the target object in the starting frame image; Based on the local key points, using the preset image segmentation model, performing target segmentation on the starting frame image to obtain a mask image corresponding to the starting frame image; Based on the mask image corresponding to the starting frame image, a segmentation result of the target object is determined.

3. A material generation method, characterized in that: include: Obtaining a segmentation result and a tracking result of a target object in a video frame sequence to be processed, wherein the segmentation result and the tracking result are determined by the target segmentation and tracking method according to claim 1; Based on the segmentation result and the tracking result, generating a transparent channel video frame sequence corresponding to the video frame sequence; Generate naked-eye 3D scene card material according to the transparent channel video frame sequence and the video frame sequence.

4. A naked eye 3D scene card material, characterized in that: The material is produced by the material production method according to claim 9.

5. A target segmentation and tracking device, characterized in that: include: An acquisition unit, used for acquiring a video frame sequence to be processed, and a positive point added by an operator for a target object on a starting frame image of the video frame sequence; A segmentation unit, configured to perform target segmentation on the starting frame image based on the forward point and a preset image segmentation model to obtain a segmentation result of the target object; The tracking unit is used to perform continuous target tracking and segmentation on each video frame image in the video frame sequence according to the segmentation result and a preset target tracking model to obtain the tracking result of the target object.

6. A material generating device, characterized in that: include: An acquisition unit, used to acquire a segmentation result and a tracking result of a target object in a video frame sequence to be processed, wherein the segmentation result and the tracking result are determined by the target segmentation and tracking method according to claim 1; A first generating unit, configured to generate a transparent channel video frame sequence corresponding to the video frame sequence based on the segmentation result and the tracking result; The second generating unit is used to generate naked-eye 3D scene card materials according to the transparent channel video frame sequence and the video frame sequence.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target segmentation and tracking method according to any one of claims 1 to 2 is implemented.

8. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the target segmentation and tracking method according to any one of claims 1 to 2 is implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the material generation method according to claim 3 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the material generation method according to claim 3 is implemented.

Images