Video target segmentation method, device and equipment and computer readable storage medium

Abstract

The application discloses a target segmentation method and device of a video, equipment and a computer readable storage medium, and belongs to the field of image processing. The method comprises the following steps: acquiring a first video frame from a video frame sequence; processing the first video frame through a curve placement segmentation network model, generating a closed curve of a target edge in the first video frame, and taking the generated closed curve as a first target edge; adjusting the position of the first target edge, and determining the target edge in the video frame after the first video frame to track the target in the subsequent video frame. The application can accurately extract the target edge from the video frame through the curve placement segmentation network model, and improves the completeness and definition of the segmented target. In addition, the position of the extracted target edge is adjusted to determine the target edge of the subsequent video frame, which improves the edge stability of the segmented target and makes the tracking effect more stable.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method, apparatus, device, and computer-readable storage medium for target segmentation in video. Background Technology

[0002] Video target segmentation refers to the technique and process of dividing a video frame into several specific regions with unique properties and extracting the target of interest from them. It is an important step in target detection.

[0003] In related technologies, semantic segmentation is typically used for target segmentation in videos, and each video frame in a video frame sequence is segmented separately using semantic segmentation. In the process of segmenting each video frame individually using semantic segmentation, each pixel in the video frame is first classified to obtain its category. Then, based on the category of each pixel, the set of pixels belonging to the target is determined from all pixels in the video frame, and this determined set of pixels is used as the target to be segmented.

[0004] However, semantic segmentation technology can only identify a set of pixels, and the pixels in this set may not be related to each other, resulting in a lack of overall integrity in the boundaries of the pixel set and an inability to segment a complete and clear target. Moreover, using semantic segmentation technology to segment each video frame separately may lead to significant variations in the same target segmented from different video frames, making the fusion of inter-frame information difficult and resulting in low stability. Summary of the Invention

[0005] This application provides a video target segmentation method, apparatus, device, and computer-readable storage medium, which can solve the problems of inability to segment complete and clear targets and low stability of video segmentation in related technologies. The technical solution is as follows:

[0006] On the one hand, a video target segmentation method is provided, including:

[0007] Obtain the first video frame from the video frame sequence, wherein the first video frame is any video frame in the video frame sequence except for the last video frame;

[0008] The first video frame is processed by a curve sculpting network model to generate a closed curve of the target edge in the first video frame, and the generated closed curve is used as the first target edge.

[0009] The position of the first target edge is adjusted to determine the target edge in the video frame following the first video frame, so as to perform target tracking on the video frame following the first video frame.

[0010] On the other hand, a video target segmentation apparatus is provided, comprising:

[0011] The first acquisition module is used to acquire a first video frame from a video frame sequence, wherein the first video frame is any video frame in the video frame sequence except for the last video frame.

[0012] The model processing module is used to process the first video frame through the curve placement segmentation network model, generate closed curves of the target edges in the first video frame, and use the generated closed curves as the first target edges;

[0013] The target tracking module is used to adjust the position of the edge of the first target and determine the target edge in the video frame after the first video frame, so as to perform target tracking on the video frame after the first video frame.

[0014] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements any of the aforementioned video target segmentation methods.

[0015] In another aspect, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements any of the above-described video target segmentation methods.

[0016] On the other hand, a computer program product is also provided, which, when executed, is used to implement any of the above-mentioned video target segmentation methods.

[0017] The beneficial effects of the technical solutions provided in this application are:

[0018] In this embodiment, a curve-swinging segmentation network model is used to process the first video frame in the video frame sequence to generate a closed curve representing the target edge in the first video frame. This generated closed curve is then used as the first target edge. The curve-swinging segmentation network model accurately extracts the target edge from the first video frame, precisely identifying the target boundary and thus improving the completeness and clarity of the segmented target. Furthermore, after extracting the first target edge from the first video frame, the position of the first target edge is adjusted to determine the target edge in subsequent video frames. This not only fuses information between frames but also improves the edge stability of the segmented target, resulting in more stable tracking performance. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a target segmentation system based on a curve-swinging segmentation network model provided in an embodiment of this application;

[0021] Figure 2 This is a flowchart of a video target segmentation method provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of video segmentation provided in an embodiment of this application;

[0023] Figure 4 This is a schematic diagram illustrating the processing procedure of an edge matching model provided in an embodiment of this application;

[0024] Figure 5 This is a block diagram of a video target segmentation device provided in an embodiment of this application;

[0025] Figure 6 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0027] Before providing a detailed explanation of the embodiments of this application, the application scenarios of these embodiments will be described first.

[0028] The video target segmentation method provided in this application is used to segment specific targets from video frames and to track those targets. The target to be segmented can be configured as needed, such as a person, vehicle, or road. For example, the method provided in this application can be applied to the fields of autonomous driving, traffic monitoring, video surveillance, or face tracking, and of course, it can also be applied to other related fields; this application does not limit its application to these fields.

[0029] The method provided in this application can be applied to computer devices such as terminal devices or servers. The terminal device can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. This application does not limit the specific type of terminal device.

[0030] It should be noted that the video target segmentation method provided in this application embodiment is implemented using a target segmentation system based on a curve oscillation segmentation network model. For ease of understanding, the target segmentation system will be introduced below.

[0031] Figure 1 This is a schematic diagram of a target segmentation system based on a curve-arranged segmentation network model provided in an embodiment of this application, as shown below. Figure 1 As shown, the system includes an image to be segmented 11, an edge curve map 12, a stitching 13, and a curve-positioned segmentation network model 14.

[0032] Among them, the curve-positioning segmentation network model 14 is used to extract the target edge in the image to be segmented 11, that is, to generate the closed curve of the target edge in the image to be segmented 11, and to use the generated closed curve as the target edge.

[0033] For example, the curve sway segmentation network model 14 sequentially predicts the anchor points and curvatures on the target edges in the image to be segmented 11, and generates the target edges based on the sequentially predicted anchor points and curvatures. The input of the curve sway segmentation network model 14 is the image to be segmented 11, and the initial or currently generated edge curve map 12, and the output is the position of the next anchor point and the corresponding curvature of the generated partial target edges.

[0034] Before performing target segmentation on the image 11 to be segmented, the edge curve graph 12 may have initial anchor points of the target edges in the image 11 to be segmented, so that the curve swaying segmentation network model 14 can predict the position and corresponding curvature of the next anchor point based on the initial anchor points.

[0035] The initial anchor points on the image 11 to be segmented can be preset, such as manually or randomly, or they can be predicted by the curve placement segmentation network model 14 using the initial values. This application embodiment does not limit this.

[0036] The stitching step 13 is used to stitch the image to be segmented 11 with the edge curve map 12. Each time a prediction is performed, the image to be segmented 11 and the edge curve map 12 need to be stitched together first, and then input into the curve placement segmentation network model 14 for processing.

[0037] After predicting the position and corresponding curvature of the next anchor point of the partially generated edge curve in edge curve map 12 based on the image to be segmented 11 and edge curve map 12, the curve sizing segmentation network model 14 can place the next anchor point onto the partially generated edge curve in edge curve map 12 to extend the generated edge curve in edge curve map 12. Then, based on the newly generated edge curve in edge curve map 12 and the image to be segmented 11, the curve sizing segmentation network model 14 continues to predict the position and corresponding curvature of the next anchor point. This process is repeated until a closed curve is generated based on the sequentially predicted anchor points and curvatures. The generated closed curve is then used as the target edge of the image to be segmented 11.

[0038] Figure 2 This is a flowchart of a video target segmentation method provided in an embodiment of this application. This method is applied in computer devices, such as... Figure 2 As shown, the method includes the following steps:

[0039] Step 201: Obtain the first video frame from the video frame sequence. The first video frame is any video frame in the video frame sequence except for the last video frame.

[0040] The video frame sequence comprises multiple video frames, which can be a sequence of video frames from the target video to be segmented. The first video frame can be the first video frame or an intermediate video frame in the video frame sequence, etc.

[0041] The target video can be obtained by capturing it with a camera, retrieving it from storage space, downloading it from the network, or sending it from other devices; this application embodiment does not limit this. The target video can be autonomous driving video, traffic monitoring video, or other videos, etc.

[0042] Step 202: Process the first video frame using the curve placement segmentation network model to generate a closed curve of the target edge in the first video frame, and use the generated closed curve as the first target edge.

[0043] In this context, the target edge in the first video frame refers to the edge of the target to be segmented in the first video frame. The first target edge is the closed curve at the target edge extracted from the first video frame by the curve placement segmentation network model, used to indicate the segmented target.

[0044] As an example, processing the first video frame using a curve sway segmentation network model to generate a closed curve of the target edge in the first video frame includes: processing the first video frame using a curve sway segmentation network model, sequentially predicting the anchor points and curvature on the target edge in the first video frame, generating a closed curve based on the sequentially predicted anchor points and curvature, and using the generated closed curve as the first target edge.

[0045] As an example, we can first determine the initial anchor point on the target edge in the first video frame. Then, based on the initial anchor point and the first video frame, a curve swaying segmentation network model predicts the next anchor point and its corresponding curvature. The predicted anchor point and curvature are then combined with the initial anchor point to generate a partial edge curve. Next, based on the newly generated edge curve and the first video frame, the curve swaying segmentation network model continues to predict the next anchor point and its corresponding curvature. The predicted anchor point and curvature are then combined with the already generated edge curve to extend it. This process is repeated until a closed curve is generated based on the sequentially predicted anchor points and curvatures. The generated closed curve is then used as the target edge of the first video frame, resulting in the first target edge.

[0046] By processing the first video frame using a curve-aligned segmentation network model, the target edges in the first video frame can be extracted. The extracted target edges can be used to circle the target to be segmented, making it easier to segment out a complete and clear target.

[0047] It should be noted that, in this embodiment of the application, in order to predict the next anchor point and curvature of the edge curve, the curve swaying segmentation network model needs to be trained in advance based on sample data. For example, edge extraction can be performed on multiple sample images to obtain the edge curve of each sample. Then, based on the multiple sample images and the corresponding edge images, the curve swaying segmentation network model to be trained is trained to obtain the trained curve swaying segmentation network model.

[0048] Step 203: Adjust the position of the first target edge to determine the target edge in the video frame after the first video frame, so as to perform target tracking on the video frame after the first video frame.

[0049] In other words, after obtaining the first target edge in the first video frame, the first target edge can be fine-tuned in subsequent video frames to determine the target edge in each subsequent video frame, thereby completing video segmentation. This improves the stability of segmentation and allows for better tracking of the segmented region.

[0050] The video frames following the first video frame include n video frames following the first video frame, where n is a positive integer. n can be a pre-set value, such as 1, 2, or 5. Alternatively, n can be the total number of video frames following the first video frame; that is, the video frames following the first video frame include all video frames following the first video frame. This embodiment of the application does not limit this. For example, the video frames following the first video frame include the next video frame after the first video frame, and any video frame after the next video frame.

[0051] As an example, adjusting the position of the first target edge to determine the target edge in video frames following the first video frame can include the following two scenarios:

[0052] The first scenario: For the second video frame, the position of the first target edge is adjusted to obtain the second target edge, which is then defined as the target edge of the second video frame. Here, the second video frame refers to the video frame following the first video frame.

[0053] In other words, for the next video frame after the first video frame, the position of the first target edge can be adjusted in the next video frame to determine the target edge in the next video frame.

[0054] Please refer to Figure 3 , Figure 3 This is a schematic diagram of video segmentation provided in an embodiment of this application. Figure 3 The left image in the image is the first video frame. Figure 3 The middle image in the image is the second video frame. After segmenting the first video frame using the curve-swinging segmentation network model, a target edge 30 can be obtained, with the target being an airport terminal sign. Then, based on the position of target edge 30, it can be drawn onto the second video frame, as shown below. Figure 3 As shown in the middle image. (From...) Figure 3 As shown in the middle image, target edge 30 does not perfectly match the target, but the difference is small. Therefore, the position of target edge 30 can be adjusted in the second video frame to make the adjusted target edge 30 match the target. The adjustment process can be as follows: Figure 3 As shown in the right figure.

[0055] In the second video frame, the adjustment granularity for adjusting the position of the first target edge can be the entire first target edge, or it can be anchor points or line segments within the first target edge; this embodiment does not limit this. That is, the first target edge can be translated as a whole, or it can be adjusted according to anchor point granularity or line segment granularity, such as moving anchor points or line segments within the first target edge individually. Furthermore, the translation or movement can be performed in any direction; this embodiment does not limit this.

[0056] By translating the entire edge of the first target, the overall position of the first target edge can be moved; by adjusting the first target edge according to the anchor point granularity or line segment granularity, not only can the overall position of the first target edge be moved, but also the deformation of the first target edge can be achieved, such as enlarging or shrinking.

[0057] As an example, adjusting the position of the first target edge in the second video frame to obtain the second target edge may include the following steps:

[0058] 1) In the second video frame, the position of the edge of the first target is adjusted according to the preset rules.

[0059] In the second video frame, adjusting the position of the edge of the first target according to preset rules can include the following two implementation methods:

[0060] The first implementation method is to adjust the position of the edge of the first target in the second video frame according to a random adjustment mechanism.

[0061] As an example, the position of the first target edge can be translated according to a random translation mechanism. For instance, the position of the first target edge can be translated by a target length in a random translation direction. This random translation direction can be determined by a random algorithm. The target length can be preset by default, determined based on the size of the first target edge, or generated by a random algorithm; this embodiment does not limit this. For example, the target length can be 1 / n of the edge length of the first target edge, where n is a positive integer.

[0062] As another example, the edge of the first target can be randomly adjusted at the anchor point granularity or the line segment granularity. For instance, the anchor points or line segments in the edge of the first target can be adjusted according to a random adjustment mechanism. The anchor points or line segments in the edge of the first target that need adjustment can be determined by a random algorithm. Alternatively, the anchor points or line segments in the edge of the first target can first be deformed according to a random deformation mechanism, such as enlarging or shrinking, and then the deformed edge of the first target can be translated according to a random translation mechanism.

[0063] Of course, other methods can also be used to adjust the position of the first target edge according to the random adjustment mechanism, and this application embodiment does not limit this.

[0064] Furthermore, in the second video frame, after randomly adjusting the position of the first target edge, it can be determined whether the adjusted position of the first target edge matches the target in the second video frame. If they match, the adjusted first target edge is used as the second target edge. If they do not match, the position of the first target edge is randomly adjusted again until the adjusted first target edge matches the target in the second video frame.

[0065] The second implementation method is as follows: Based on the temporal changes of pixels in the first and second video frames, and the correlation between the first and second video frames, calculate the motion vector of optical flow between the first and second video frames; and adjust the position of the edge of the first target based on the motion vector.

[0066] The motion vector of the optical flow can be used to indicate the distance each point moves on the first video frame on the second video frame. Therefore, on the second video frame, the position of the first target edge can be adjusted based on the motion vector to match the actual target.

[0067] Please refer to Figure 3 The right image in the middle, Figure 3 The arrows in the right-hand diagram indicate the motion vector of the optical flow. By adjusting the position of the first target edge according to this motion vector, the adjusted first target edge can be matched with the target.

[0068] 2) Input the second video frame and the adjusted first target edge into the target edge matching model, and determine the matching information between the adjusted first target edge and the target in the second video frame through the edge matching model.

[0069] The matching information indicates whether the adjusted first target edge matches the target in the second video frame. The edge matching model is a binary classification network model used to identify whether the input target edge matches the target in the video frame. Matching the target edge with the target in the video frame means that the target edge has a high degree of overlap with the true edge of the target in the video frame, for example, the overlap degree is greater than or equal to the overlap degree threshold.

[0070] The edge matching model takes a target edge and a video frame as input and outputs matching information. This matching information can be a matching probability, which indicates the likelihood of a match between the input target edge and a target in the input video frame. For example, if the matching probability is greater than or equal to a probability threshold, a match is determined; if the matching probability is less than the probability threshold, a mismatch is determined. This probability threshold can be preset, for example, it could be 0.8 or 0.9.

[0071] Please refer to Figure 4 When image 41 is input into edge matching model 43, the obtained matching information can indicate that the segmented target edge does not match the target in image 41; when image 42 is input into edge matching model 43, the obtained matching information can indicate that the segmented target edge matches the target in image 42.

[0072] The edge matching model can be trained in advance using sample data.

[0073] As an example, you can obtain sample data, and then train the target edge matching model to be trained based on the sample data to obtain the trained target edge matching model.

[0074] The sample data may include multiple positive sample data, each positive sample data including a positive sample image and a corresponding target edge, wherein the target edge corresponding to the positive sample image matches the target in the positive sample image. Furthermore, the sample data may also include multiple positive sample data and multiple negative sample data, each negative sample data including a negative sample image and a corresponding target edge, wherein the target edge corresponding to the negative sample image does not match the target in the negative sample image.

[0075] After determining the matching information between the adjusted first target edge and the target in the second video frame using an edge matching model, the second target edge can be determined based on the matching information. The process of determining the second target edge based on the matching information may include the following steps 3)-4).

[0076] 3) If the adjusted first target edge is determined to match the target in the second video frame based on the matching information, then the adjusted first target edge is determined as the second target edge.

[0077] 4) If the adjusted first target edge does not match the target in the second video frame based on the matching information, the position of the adjusted first target edge is adjusted according to the preset rules until the adjusted first target edge matches the target in the second video frame, and the adjusted first target edge is determined as the second target edge.

[0078] That is, if the adjusted first target edge does not match the target in the second video frame based on the matching information, return to step 1) and continue to adjust the position of the adjusted first target edge according to the preset rules. Then, input the second video frame and the adjusted first target edge into the target edge matching model. The matching information between the adjusted first target edge and the target in the second video frame is determined by the edge matching model. The adjustment stops when the adjusted first target edge matches the target in the second video frame based on the matching information, and the adjusted first target edge is determined as the second target edge.

[0079] In addition, the target edges extracted by the curve sculpting network model may result in too many target circles, too few target circles, or exactly the target circled.

[0080] For situations with multiple target circles, the targets within the target edge can be further subdivided, dividing the current target into multiple sub-targets, i.e., internal targets. Then, the movement of the edges of these internal targets is further analyzed. If the analysis determines that the edge of a sub-target within the internal target is relatively stationary compared to the edges of other targets, then this sub-target can be considered part of the current target. If the analysis determines that the edge of a sub-target within the internal target is relatively moving compared to the edges of other targets, then this sub-target can be identified as a circled target and removed from the current target set.

[0081] As an example, a curve-swinging segmentation network model can be used to process the first video frame, sequentially predicting the anchor points and curvatures on the sub-target edges within the first target edge. Based on these predicted anchor points and curvatures, closed curves are generated, and these closed curves are used as sub-target edges, resulting in multiple sub-target edges. Then, based on these multiple sub-target edges, target tracking is performed on video frames following the first video frame. If, according to the target tracking results, the first sub-target edge among the multiple sub-target edges is determined to be moving relative to the other sub-target edges, then the first sub-target edge is separated from the first target edge. In this way, the extra-circled targets can be removed from the current target pool. Here, the first sub-target edge can be any one of the multiple sub-target edges.

[0082] As an example, the motion of a first sub-target relative to other sub-targets can be determined by the fact that the edge of the first sub-target is moving relative to the edges of other sub-targets.

[0083] In the case of fewer circles, it can be determined whether the current target is relatively stationary (cooperative movement) with other targets. If the current target is relatively stationary with other targets, then the current target and other targets can be treated as the same target and merged into one target, so as to merge the target with fewer circles into the current target.

[0084] As an example, if the first target edge comprises multiple target edges, target tracking can be performed on video frames subsequent to the first video frame based on the positions of these multiple target edges. If, based on the tracking results, it is determined that one of these multiple target edges is relatively stationary compared to any other target edge, then this target edge can be merged with any other target edge into a single target edge. In this way, targets with fewer loops can be merged into the current target.

[0085] As another example, a curve-segmentation network model can be used to segment the region outside the second target edge in the second video frame to obtain the third target edge; then, based on the second and third target edges, target tracking can be performed on the video frames following the second video frame; if it is determined from the target tracking results that the second and third target edges are relatively stationary, then the second and third target edges are merged into one target edge.

[0086] In this way, newly appearing targets with fewer circles can be merged into the current target. For example, targets that were occluded and then reappeared can be merged with the current target.

[0087] The second scenario: For the third video frame, the position of the specified target edge is adjusted within the third video frame to obtain a fourth target edge, which is then designated as the target edge of the third video frame. Here, the third video frame is any video frame following the second video frame, and the specified target edge is the target edge of the video frame preceding the third video frame.

[0088] In other words, for any video frame after the second video frame, the target edge in the current video frame can be determined by adjusting the position of the target edge in the previous video frame. Additionally, the target edge can also be determined by adjusting the position of the target edge in the video frame two frames prior to the third video frame.

[0089] It should be noted that adjusting the position of the specified target edge in the third video frame to obtain the fourth target edge and determining the fourth target edge as the target edge of the third video frame is similar to adjusting the position of the first target edge in the second video frame to obtain the second target edge and determining the second target edge as the target edge of the second video frame. The specific implementation method can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0090] In this embodiment, a curve-swinging segmentation network model is used to process the first video frame in the video frame sequence to generate a closed curve representing the target edge in the first video frame. This generated closed curve is used as the first target edge, enabling accurate extraction of the target edge from the first video frame—that is, accurately extracting the target boundary—thereby improving the completeness and clarity of the segmented target. Furthermore, after extracting the first target edge from the first video frame, the position of the first target edge is adjusted to determine the target edge in subsequent video frames. This not only fuses information between frames but also improves the edge stability of the segmented target, resulting in more stable tracking performance.

[0091] In addition, by processing the first video frame in the video frame sequence using the curve sway segmentation network model, the anchor points and curvature on the target edge in the first video frame are predicted sequentially. Based on the sequentially predicted anchor points and curvature, a closed curve is generated. The generated closed curve is used as the first target edge. By combining the prediction of edge anchor points and curvature, the edge of the target can be accurately extracted from the first video frame, which improves the accuracy and completeness of target segmentation.

[0092] In addition, since the target edges extracted in this embodiment are holistic, and the target segmentation and tracking of subsequent video frames are also based on the target edges extracted in the current video frame, the original shape of the target can still be preserved even if the target is partially occluded in the future, and there will be no tracking loss.

[0093] Figure 5 This is a block diagram of a video target segmentation device provided in an embodiment of this application. This device can be integrated into a computer device, such as... Figure 5 As shown, the device may include:

[0094] The first acquisition module 501 is used to acquire a first video frame from a video frame sequence, wherein the first video frame is any video frame in the video frame sequence except for the last video frame.

[0095] The model processing module 502 is used to process the first video frame through the curve sculpting network model, generate a closed curve of the target edge in the first video frame, and use the generated closed curve as the first target edge.

[0096] The target tracking module 503 is used to adjust the position of the edge of the first target and determine the target edge in the video frame after the first video frame, so as to perform target tracking on the video frame after the first video frame.

[0097] Optionally, the model processing module 502 is used for:

[0098] The first video frame is processed by a curve sway segmentation network model to predict the anchor points and curvature on the target edges in the first video frame in turn.

[0099] Based on the sequentially predicted anchor points and curvature, a closed curve of the target edge in the first video frame is generated.

[0100] Optionally, the target tracking module 503 includes:

[0101] The first adjustment unit is used to adjust the position of the first target edge in the second video frame to obtain the second target edge, and to determine the second target edge as the target edge of the second video frame, wherein the second video frame is the next video frame after the first video frame.

[0102] Optionally, the target tracking module 503 includes:

[0103] The second adjustment unit is used to adjust the position of the specified target edge in the third video frame to obtain the fourth target edge, and to determine the fourth target edge as the target edge of the third video frame. The third video frame is any video frame located after the second video frame, and the specified target edge is the target edge of the video frame preceding the third video frame.

[0104] Optionally, the first adjustment unit is used for:

[0105] In the second video frame, the position of the edge of the first target is adjusted according to preset rules;

[0106] The second video frame and the adjusted first target edge are input into the target edge matching model. The matching information between the adjusted first target edge and the target in the second video frame is determined by the edge matching model. The second target edge is then determined based on the matching information.

[0107] Optionally, the first adjustment unit is used for:

[0108] In the second video frame, the position of the edge of the first target is randomly adjusted;

[0109] or,

[0110] Based on the temporal changes of pixels in the first and second video frames, and the correlation between the first and second video frames, the motion vector of optical flow between the first and second video frames is calculated; the position of the edge of the first target is adjusted based on the motion vector.

[0111] Optionally, the first adjustment unit is used for:

[0112] If the adjusted first target edge is determined to match the target in the second video frame based on the matching information, then the adjusted first target edge is determined as the second target edge;

[0113] If, based on the matching information, it is determined that the adjusted first target edge does not match the target in the second video frame, then the position of the adjusted first target edge is adjusted according to the preset rule until it is determined that the adjusted first target edge matches the target in the second video frame, and the adjusted first target edge is determined as the second target edge.

[0114] Optionally, the device further includes:

[0115] The second acquisition module is used to acquire sample data, which includes multiple positive sample data; wherein each positive sample data includes a positive sample image and a corresponding target edge, and the target edge corresponding to the positive sample image matches the target in the positive sample image;

[0116] The first training module is used to train the target edge matching model to be trained based on the multiple sample data, so as to obtain the trained target edge matching model.

[0117] Optionally, the sample data also includes multiple negative sample data, each negative sample data including a negative sample image and a corresponding target edge, wherein the target edge corresponding to the negative sample image does not match the target in the negative sample image.

[0118] Optionally, the device also includes a merging module;

[0119] The model processing module 502 is also used to perform target segmentation on the region outside the second target edge in the second video frame through the curve placement segmentation network model to obtain the third target edge;

[0120] The target tracking module 503 is also used to perform target tracking on video frames located after the second video frame based on the second target edge and the third target edge;

[0121] The merging module is used to merge the edge of the second target with the edge of the third target if it is determined from the target tracking results that the edge of the second target and the edge of the third target are relatively stationary.

[0122] Optionally, the device also includes a separation module;

[0123] The model processing module 502 is also used to process the first video frame through the curve placement segmentation network model, predict the anchor points and curvature on the sub-target edges in the first target edge in turn, generate closed curves based on the predicted anchor points and curvatures in turn, and use the generated closed curves as sub-target edges to obtain multiple sub-target edges;

[0124] The target tracking module 503 is also used to perform target tracking on video frames located after the first video frame based on the edges of the multiple sub-targets;

[0125] The separation module is used to separate the first sub-target edge from the first target edge if it is determined from the target tracking result that the first sub-target edge among the plurality of sub-target edges is moving relative to the other sub-target edges. The first sub-target edge is any one of the plurality of sub-target edges.

[0126] Optionally, the device further includes:

[0127] The extraction module is used to extract edges from multiple sample images separately to obtain the edge curve of each sample;

[0128] The second training module is used to train the curve sway segmentation network model to be trained based on the multiple sample images and the corresponding edge images, so as to obtain the trained curve sway segmentation network model.

[0129] In this embodiment, a curve-swinging segmentation network model is used to process the first video frame in the video frame sequence to generate a closed curve representing the target edge in the first video frame. This generated closed curve is used as the first target edge, enabling accurate extraction of the target edge from the first video frame—that is, accurately extracting the target boundary—thereby improving the completeness and clarity of the segmented target. Furthermore, after extracting the first target edge from the first video frame, the position of the first target edge is adjusted to determine the target edge in subsequent video frames. This not only fuses information between frames but also improves the edge stability of the segmented target, resulting in more stable tracking performance.

[0130] In addition, since the target edges extracted in this embodiment are holistic, and the target segmentation and tracking of subsequent video frames are also based on the target edges extracted in the current video frame, the original shape of the target can still be preserved even if the target is partially occluded in the future, and there will be no tracking loss.

[0131] Figure 6 This is a structural block diagram of a computer device 600 provided in an embodiment of this application. The computer device 600 can be an electronic device such as a mobile phone, tablet computer, smart TV, multimedia playback device, wearable device, desktop computer, or server. The computer device 600 can be used to implement the video target segmentation method provided in the above embodiments.

[0132] Typically, computer device 600 includes a processor 601 and a memory 602.

[0133] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0134] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one instruction, which is executed by the processor 601 to implement the video target segmentation method provided in the method embodiments of this application.

[0135] In some embodiments, the computer device 600 may optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device may include at least one of a display screen 604, audio circuitry 605, communication interface 606, and power supply 607.

[0136] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the computer device 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0137] In an exemplary embodiment, a computer-readable storage medium is also provided, on which instructions are stored, which, when executed by a processor, implement the above-described video target segmentation method.

[0138] In an exemplary embodiment, a computer program product is also provided, which, when executed, is used to implement the target segmentation method for the video described above.

[0139] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0140] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0141] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for target segmentation in video, characterized in that, include: Obtain a first video frame from the video frame sequence, wherein the first video frame is any video frame in the video frame sequence except for the last video frame; The first video frame is processed by a curve sway segmentation network model to predict the anchor points and curvature on the target edge in the first video frame in turn. Based on the predicted anchor points and curvature, a closed curve of the target edge in the first video frame is generated and the generated closed curve is used as the first target edge. The position of the first target edge is adjusted to determine the target edge in the video frame after the first video frame, so as to perform target tracking on the video frame after the first video frame.

2. The method as described in claim 1, characterized in that, The step of adjusting the position of the first target edge to determine the target edge located in a video frame following the first video frame includes: For the second video frame, the position of the first target edge is adjusted in the second video frame to obtain the second target edge, and the second video frame is the next video frame after the first video frame; The second target edge is determined as the target edge of the second video frame.

3. The method as described in claim 1, characterized in that, The step of adjusting the position of the first target edge to determine the target edge located in a video frame following the first video frame includes: For the third video frame, the position of the specified target edge is adjusted in the third video frame to obtain the fourth target edge. The third video frame is any video frame located after the second video frame, and the specified target edge is the target edge of the video frame preceding the third video frame. The fourth target edge is determined as the target edge of the third video frame.

4. The method as described in claim 2, characterized in that, The step of adjusting the position of the first target edge in the second video frame to obtain the second target edge includes: In the second video frame, the position of the edge of the first target is adjusted according to a preset rule; The second video frame and the adjusted first target edge are input into the target edge matching model. The matching information between the adjusted first target edge and the target in the second video frame is determined by the edge matching model. The second target edge is determined based on the matching information.

5. The method as described in claim 4, characterized in that, The step of adjusting the position of the edge of the first target in the second video frame according to a preset rule includes: Based on the temporal changes of pixels in the first and second video frames, and the correlation between the first and second video frames, the motion vector of optical flow between the first and second video frames is calculated; the position of the first target edge is adjusted based on the motion vector.

6. The method as described in claim 4, characterized in that, The step of adjusting the position of the edge of the first target in the second video frame according to a preset rule includes: In the second video frame, the position of the edge of the first target is adjusted according to a random adjustment mechanism.

7. The method as described in claim 4, characterized in that, Determining the edge of the second target based on the matching information includes: If the adjusted first target edge is determined to match the target in the second video frame based on the matching information, then the adjusted first target edge is determined as the second target edge; If, based on the matching information, it is determined that the adjusted first target edge does not match the target in the second video frame, then the position of the adjusted first target edge is adjusted according to the preset rules until it is determined that the adjusted first target edge matches the target in the second video frame, and the adjusted first target edge is determined as the second target edge.

8. The method as described in claim 4, characterized in that, Before inputting the second video frame and the adjusted first target edge into the target edge matching model, the method further includes: Acquire sample data, which includes multiple positive sample data, each positive sample data including a positive sample image and a corresponding target edge, and the target edge corresponding to the positive sample image matches the target in the positive sample image; Based on the sample data, the target edge matching model to be trained is trained to obtain the trained target edge matching model.

9. The method as described in claim 8, characterized in that, The sample data also includes multiple negative sample data, each of which includes a negative sample image and a corresponding target edge, and the target edge corresponding to the negative sample image does not match the target in the negative sample image.

10. The method as described in claim 2, characterized in that, After determining the second target edge as the target edge of the second video frame, the method further includes: Using the curve-swing segmentation network model, target segmentation is performed on the region outside the second target edge in the second video frame to obtain the third target edge; Based on the second target edge and the third target edge, target tracking is performed on video frames located after the second video frame; If the target tracking results determine that the edge of the second target and the edge of the third target are relatively stationary, then the edge of the second target and the edge of the third target are merged.

11. The method according to any one of claims 1-10, characterized in that, After using the generated closed curve as the first target edge, the method further includes: The first video frame is processed by the curve sculpting segmentation network model to generate closed curves of sub-target edges in the first target edge. The generated closed curves are used as sub-target edges to obtain multiple sub-target edges. Based on the multiple sub-target edges, target tracking is performed on video frames located after the first video frame; If, based on the target tracking results, it is determined that the first sub-target edge among the plurality of sub-target edges is moving relative to the other sub-target edges, then the first sub-target edge is separated from the first target edge, wherein the first sub-target edge is any one of the plurality of sub-target edges.

12. The method according to any one of claims 1-10, characterized in that, Before processing the first video frame using the curve sway segmentation network model, the process further includes: Edge extraction is performed on multiple sample images to obtain the edge curve of each sample; Based on the multiple sample images and the corresponding edge images, the curve sway segmentation network model to be trained is trained to obtain the trained curve sway segmentation network model.

13. A target segmentation device for video, characterized in that, include: The first acquisition module is used to acquire a first video frame from a video frame sequence, wherein the first video frame is any video frame in the video frame sequence except for the last video frame. The model processing module is used to process the first video frame through the curve sway segmentation network model, predict the anchor points and curvature on the target edge in the first video frame in sequence, generate the closed curve of the target edge in the first video frame based on the predicted anchor points and curvature, and use the generated closed curve as the first target edge. The target tracking module is used to adjust the position of the first target edge and determine the target edge in the video frame after the first video frame, so as to perform target tracking on the video frame after the first video frame.

14. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 12.

Images