Video instance segmentation method and related device

Abstract

The present application provides a video instance segmentation method and related devices, the method comprising: obtaining a target video, wherein the target video is obtained by using a video acquisition device to capture multiple targets in a preset area; obtaining multiple detection frames for each frame of the target video, wherein each target corresponds to one or more detection frames; using the multiple detection frames for each frame, obtaining multiple sub-images corresponding to each frame, wherein each target corresponds to one sub-image; using an instance segmentation model to segment the targets in each sub-image corresponding to each frame, to obtain a segmentation result for each target corresponding to each frame. When segmenting each target, only the partial image (i.e., sub-image) corresponding to a single target can be segmented. Compared with segmenting each target from the entire image, this can reduce the amount of computation and improve the efficiency of video instance segmentation; each target in the video can be quickly located and segmented in complex scenes.

Description

Technical Field

[0001] The present application relates to the technical fields of artificial intelligence, image processing, and computer vision, and in particular to a video instance segmentation method and related devices. Background Art

[0002] In recent years, with the development of multimedia and internet technologies, video has become increasingly important as a medium for information transmission in social life. Image segmentation based on video sequences has begun to gain popularity and has played a crucial role in video coding, anomaly detection, and target monitoring. Over the past few decades, numerous researchers at home and abroad have proposed numerous algorithms for video segmentation from diverse perspectives. These algorithms can be categorized as automatic or semi-automatic, depending on whether or not they require human guidance. Recent research indicates that current video segmentation algorithms primarily focus on low-level image attributes such as texture, color, and contours, while paying little attention to the spatial motion characteristics of objects and the correlation between multiple objects. Based on current research, traditional segmentation algorithms are now focusing on the correlation between multiple objects, incorporating techniques such as superpixel segmentation, moving object detection, and morphological filtering.

[0003] In his master's thesis, "Research on Video Segmentation Algorithms" (Master's Electronic Journal Publication Information: Issue 3, 2016), Kangrui Chen proposed a new unsupervised automatic video segmentation algorithm. The algorithm uses moving object detection technology to extract the moving foreground of video images and superpixel segmentation technology to represent this moving foreground, reducing the image data dimensionality and, in turn, the time and space complexity of subsequent algorithms, ensuring good real-time performance. The concepts of connection and matching are introduced to describe the object correlation between superpixels within and between frames. This is used to establish a superpixel connection weight model, transforming the video segmentation problem into a superpixel connection weight calculation problem. Once the connection weight model is established, the algorithm calculates the connection weights using static features of the current frame and correlation features of the previous and next frames. The static features are primarily derived from the spatial positional relationships of superpixels within a frame, while the correlation features require the inclusion of temporal information from the video sequence. To optimize the efficiency of connection weight calculation, the algorithm utilizes color features between superpixels within a frame and motion correlation features between superpixels between frames to constrain the matching space to be searched, thereby improving the search speed. To validate the algorithm's effectiveness, video segmentation experiments were conducted on both simple and complex scene video sequences. In simple scenes, the algorithm achieved high recall and stable accuracy. In complex scenes, the algorithm was able to segment individual individuals in some crowds. Extensive experimental results demonstrate that the algorithm can segment video images and effectively address over-segmentation issues.

[0004] Yihang Bo published a paper titled "Rotation- and Scale-Invariant Human Segmentation in Videos" in the October 2017 issue of Acta Automatica Sinica. The paper proposes a rotation- and scale-invariant video segmentation method for human body part regions. This method considers not only the relationship between the torso and limbs, but also the relationships between the limbs. It optimizes possible combinations of body parts in each frame using spatial and temporal continuity constraints. It also cleverly uses dynamic programming to optimize a nonlinear graphical model, which is robust to scale variations and various flipping motions. The method first uses dynamic programming to obtain the optimal N body part combinations in each frame. Each combination is then treated as a node in the graphical model. Dynamic programming is then used to optimize the grid-like graph structure constructed from these combinations across all frames, ultimately yielding the optimal body part combination in each frame. Experimental results demonstrate that this video segmentation method is applicable not only to pedestrian videos but also to motion videos with various poses, with good robustness.

[0005] Although many current video object segmentation methods can improve instance segmentation results to a certain extent, they fail to solve the problems of difficult detection and segmentation of target objects in complex scenes and poor segmentation results. Therefore, the present invention proposes a new video instance segmentation algorithm. Summary of the Invention

[0006] The purpose of this application is to provide a video instance segmentation method and related devices for performing instance segmentation on multiple targets in a video, and quickly locating and segmenting each target in the video in complex scenes.

[0007] The purpose of this application is achieved by the following technical solutions:

[0008] In a first aspect, the present application provides a video instance segmentation method, the method comprising:

[0009] Acquire a target video, where the target video is obtained by capturing multiple targets in a preset area using a video capture device;

[0010] Obtain multiple detection frames for each frame of the target video, wherein each target corresponds to one or more detection frames;

[0011] Using multiple detection boxes in each frame, multiple sub-images corresponding to each frame are obtained, where each target corresponds to a sub-image.

[0012] The instance segmentation model is used to segment the target in each sub-image corresponding to each frame image to obtain the segmentation result of each target corresponding to each frame image.

[0013] The beneficial effect of this technical solution is that: a video capture device is used to capture multiple targets in a preset area to obtain a target video, and for each frame of the target video, multiple detection frames are set to locate each target in the image, and the sub-image corresponding to each target is obtained by using these detection frames. The target in each sub-image is segmented by using an instance segmentation model to obtain the segmentation result of each target. Since each target corresponds to one or more detection frames, the sub-image corresponding to each target is obtained by the detection frames. When segmenting each target, only the partial image (i.e., sub-image) corresponding to the single target can be segmented. Compared with segmenting each target from the entire image (i.e., the original image in the target video), the amount of computation can be reduced and the efficiency of video instance segmentation can be improved. On the other hand, since the detection frame of each target is first located, the single target is easy to segment. Therefore, each target in the video can be quickly located and segmented in complex scenes, and the moving target can be automatically and effectively segmented from the image sequence corresponding to the video in real time, thereby improving the effectiveness and accuracy of video instance segmentation and solving the problem of poor segmentation effect of single targets.

[0014] In some optional implementations, obtaining multiple detection frames for each frame of the target video includes:

[0015] The frame difference method is used to obtain multiple target images in each frame;

[0016] Based on multiple target images in each frame image, multiple detection frames are obtained for each frame image.

[0017] The beneficial effect of this technical solution is that: first, a plurality of target images in each frame image are obtained by using the frame difference method, and then a detection frame is set for the target image of each frame image to obtain multiple detection frame information for a single frame image. On the one hand, first obtaining the target image in each frame image, and then using the target image to set the detection frame means that setting the detection frame is divided into two steps to achieve, and an additional target image is added as the intermediate information between each frame image (i.e., the original image) and the detection frame (position information). The target image is part of the original image. Compared with setting the detection frame using the original image, setting the detection frame using the target image reduces the amount of computation; on the other hand, current video segmentation algorithms mainly focus on low-level attributes of the image such as texture, color, and contour, but pay less attention to the spatial motion characteristics of objects and the correlation analysis between multiple objects. For the video captured by the video acquisition device, the corresponding image sequence has the characteristics of continuity. If there is no moving target in the scene, the change between consecutive frames is very weak. If there is a moving target, there will be obvious changes between consecutive frames. The frame difference method is used to obtain the target image and analyze the spatial motion characteristics of the target and the correlation between multiple targets. It can make good use of the spatial motion characteristics of the target and the inherent connection between multiple targets, and solve the problem of difficult detection of moving targets due to mutual occlusion or irregular motion of multiple targets.

[0018] In some optional implementations, acquiring multiple detection frames for each frame of image based on multiple target images for each frame of image includes:

[0019] The target image is updated using the edge template of each frame image to remove holes and ghosting of the target image contour;

[0020] Based on the updated multiple target images of each frame image, multiple detection frames of each frame image are obtained.

[0021] The beneficial effect of this technical solution is that multiple target images are quickly obtained through the frame difference method. At this time, the contours of the target images obtained are prone to "holes" and "ghosting", which makes it difficult to ensure their integrity, resulting in inaccurate target detection. Therefore, the target image cannot be used directly as an object template. Instead, it is necessary to combine the (spatial) edge detection algorithm to determine the exact edge of each target in the original image through edge detection to obtain an edge template in the image. The edge template is then used to update the incomplete target image obtained by the frame difference method to remove holes and ghosting in the contours of the target image, thereby making up for the shortcomings of a single algorithm (i.e., the frame difference method), thereby segmenting a complete and accurate moving target and solving the problem of inaccurate target segmentation boundaries.

[0022] In some optional implementations, the process of obtaining the edge template of each frame of image includes:

[0023] Perform edge detection on the original image of each frame to obtain edge information of each frame;

[0024] The edge information of each frame image is binarized to obtain the edge template of each frame image.

[0025] The beneficial effect of this technical solution is that: the edge detection algorithm (in the spatial domain) is used to perform edge detection on the original image of each frame of the image, the edge information of the original image is obtained as the edge information of each frame of the image, and the obtained edge information is binarized to obtain an edge template for each frame of the image. Compared with other data processing methods, the binarization has a small amount of computation, high computational efficiency, and short computation time, which can improve the efficiency of the overall video segmentation method.

[0026] In some optional implementations, the instance segmentation model adopts an HTC model.

[0027] The beneficial effects of this technical solution are as follows: the HTC model integrates the Cascade R-CNN and Mask R-CNN models, changes the box branches from parallel to serial (that is, the box branch is executed first in each stage, and the regressed detection box is then handed over to the mask branch to predict the mask), introduces information flow between mask branches (similar to the information flow between box branches in Cascade R-CNN), and introduces the semantic segmentation branch S to provide spatial context information, thereby helping the HTC model distinguish the foreground (that is, the target image) and background (the part outside the target image) of each frame.

[0028] In some optional implementations, during the training process of the HTC model, a data enhancement method combining Copy-Paste and Mixup is used to perform data enhancement on the training set.

[0029] The beneficial effects of this technical solution are: Copy-Paste refers to mixing the contents of two images together using a preset formula. This process has a high degree of randomness. This data augmentation method is robust, has a simple strategy, is easy to insert into any instance segmentation code base, and does not increase training costs or inference time; Mixup is an algorithm used in computer vision to perform mixed-class enhancement on images. It can mix images between different classes to expand the training set. The model prediction and gradient modulus of the trained model between training images are more stable; the data augmentation method that uses the fusion of the above two algorithms can, to a certain extent, take into account the advantages of both.

[0030] In some optional embodiments, the method further comprises:

[0031] Initialize multiple trajectories using multiple detection boxes in the first frame image of the target video, where each target corresponds to one trajectory;

[0032] For each frame image other than the first frame image of the target video, use the multiple detection boxes in each frame image to update the multiple trajectories to obtain the final trajectory of each target;

[0033] Among them, the update process of the multiple trajectories includes:

[0034] Perform label embedding processing on multiple sub-images corresponding to each frame image to obtain label information corresponding to each frame image, and the label information corresponding to each frame image includes embedding feature information corresponding to multiple detection boxes in each frame image;

[0035] Use the label information corresponding to the k-th frame image and the label information corresponding to the (k - 1)-th frame image to associate at least one detection box in the k-th frame image with one of the multiple trajectories respectively, so as to update the trajectory associated with at least one detection box, where 1 < k ≤ N, N is the number of frames of the target video, and N is an integer greater than 1.

[0036] The beneficial effect of this technical solution is that: Label embedding refers to converting data into a suitable feature representation, and suitable means that the specific attributes of the sample can be represented by the concept of distance. Through embedding, the original sample data can be converted into feature data suitable for machine learning. Moreover, embedding can learn a low-dimensional (about a hundred dimensions) vector that can effectively represent each pixel for each pixel, and finally use the vector or the variable score generated by the vector as the input variable for other machine learning tasks. In this way, using label embedding to achieve the trajectory tracking of each target, first perform label embedding processing on multiple sub-images of each frame image respectively. Since each sub-image corresponds to one or more detection boxes, the embedding feature information corresponding to each detection box is obtained, and the embedding feature information can be used for calculating the feature distance between adjacent frames; in the process of obtaining the final trajectory, first initialize multiple trajectories according to multiple detection boxes in the first frame image. In subsequent frame images, for each detection box in the current frame image, first try to associate the detection box with the existing trajectories according to the feature distance between adjacent frames, thereby updating the associated trajectory (that is, updating the detection box position sequence in the trajectory). Analyzing the embedding feature information of the target in adjacent frames can make good use of the temporal characteristics of the target, and the obtained associated trajectory is more reliable.

[0037] In some optional implementation manners, the step of using the label information corresponding to the k-th frame image and the label information corresponding to the (k - 1)-th frame image to associate at least one detection box in the k-th frame image with one of the multiple trajectories respectively includes:

[0038] Taking each detection frame of the k-th frame image as a target detection frame, respectively, calculating the surface distance between the embedded feature information corresponding to the target detection frame and the embedded feature information corresponding to each detection frame of the k-1-th frame image;

[0039] When the surface distance between the embedded feature information corresponding to one and only one associated detection frame in the multiple detection frames of the k-1th frame image and the embedded feature information corresponding to the target detection frame is not greater than a preset distance, the trajectory of the target corresponding to the target detection frame and the associated detection frame is associated.

[0040] This technical solution has the beneficial effect of: the surface distance between the embedded feature information corresponding to the target detection frame and the embedded feature information corresponding to each detection frame in the k-1th frame image, for example, refers to the average surface distance between the target surface point set corresponding to the target detection frame and the target surface point set corresponding to each detection frame in the k-1th frame image. When associating the target detection frame with multiple existing trajectories, the association is first performed using the surface distances between adjacent frames. If, among the multiple surface distances between the target detection frame and multiple detection frames in the previous frame image, the surface distance between one and only one associated detection frame and the target detection frame falls within a preset distance range, the target detection frame is associated with the target trajectory corresponding to the unique associated detection frame. Detection frames contain both target and non-target areas. For detection frames with a large proportion of non-target areas, there may be cases where the detection frames are located close together in the previous frame but are not the same target. Using the target surface feature distances between adjacent frames for association produces more accurate and reliable association results than association based solely on the detection frame's position information.

[0041] In some optional implementations, the updating process of the multiple trajectories further includes:

[0042] When the surface distance between the embedded feature information corresponding to any one of the multiple detection frames in the k-1th frame image and the embedded feature information corresponding to the target detection frame is greater than the preset distance, calculating the IOU between the target detection frame and the predicted detection frame of each trajectory in the kth frame image one by one;

[0043] When the IOU between one and only one associated prediction detection box and the target detection box in the prediction detection box of the k-th frame image is not greater than a preset threshold, the target detection box and the trajectory of the target corresponding to the associated prediction detection box are associated.

[0044] The beneficial effect of this technical solution is that when the surface distance association of the detection frames between adjacent frames fails, the IOU between the target detection frame and the predicted detection frame of each trajectory in the k-th frame image is used for association. The IOU calculation is simple, the amount of calculation is small, the calculation speed is fast, the calculation time is short, and the association can be achieved quickly.

[0045] In some optional implementations, performing label embedding processing on the multiple sub-images corresponding to each frame of image to obtain label information corresponding to each frame of image includes:

[0046] The MGN model is used to perform label embedding processing on multiple sub-images corresponding to each frame image to obtain the label information corresponding to each frame image;

[0047] The MGN model is obtained by training the preset Resnet50-ibn-a network using the training set.

[0048] The beneficial effects of this technical solution are: using the preset Resnet50-ibn-a network to train the MGN model, and then using the MGN model to perform label embedding processing on sub-images. In the forward propagation of the Resnet50-ibn-a network, shallow-layer features can be reused in deep layers. During the backward propagation, the gradients of the deep layers can be directly transmitted back to the shallow layers. During the error backpropagation, only the residual part needs to be propagated, thus solving the gradient vanishing (network degradation) problem. MGN (multiple granularity network) is a multi-granularity network that combines global features with multi-granularity local features. Global features are responsible for extracting the overall macroscopic features that are shared by everyone. The image is then divided into different blocks, each with different granularity, responsible for extracting features at different levels or levels. Combining global features with local features can provide rich information and details to represent the complete situation of the input image. Practice has found that as the segmentation granularity increases, the model can learn more detailed information. The MGN model is used to embed labels for multiple sub-images in each frame image to obtain the embedded feature information corresponding to each sub-image (a combination of global features and local features). Since the sub-image corresponds to one or more detection frames of the current frame image, the embedded feature information corresponding to each detection frame is obtained.

[0049] In some optional embodiments, the preset Resnet50-ibn-a network is used to perform feature extraction on the image to be processed to obtain global features and local features;

[0050] During the training process of the MGN model, Triplet loss and Softmax loss are used to calculate the loss of the extracted global features, and Softmax loss is used to calculate the loss of the extracted local features.

[0051] The beneficial effects of this technical solution are: global features are complete but rough, local features are incomplete but fine, and using Triplet loss and Softmax loss to calculate the loss of the extracted global features can take into account the globality and accuracy of the training process; using Softmax loss to calculate the loss of the extracted local features can learn local fine information and avoid the situation where local information is ignored.

[0052] In a second aspect, the present application provides a video instance segmentation device, the device comprising:

[0053] A video acquisition module is used to acquire a target video, wherein the target video is obtained by capturing multiple targets in a preset area using a video acquisition device;

[0054] A detection frame acquisition module is used to obtain multiple detection frames for each frame of the target video, wherein each target corresponds to one or more detection frames;

[0055] A sub-image acquisition module is used to obtain multiple sub-images corresponding to each frame of the image using multiple detection frames of each frame of the image, where each target corresponds to a sub-image;

[0056] The target segmentation module is used to segment the target in each sub-image corresponding to each frame image using the instance segmentation model to obtain the segmentation result of each target corresponding to each frame image.

[0057] In some optional implementations, the detection frame acquisition module includes:

[0058] A target image unit, configured to obtain multiple target images for each frame of image using a frame difference method;

[0059] The detection frame unit is used to obtain multiple detection frames for each frame of image based on multiple target images in each frame of image.

[0060] In some optional implementations, the detection frame unit is used to:

[0061] The target image is updated using the edge template of each frame image to remove holes and ghosting of the target image contour;

[0062] Based on the updated multiple target images of each frame image, multiple detection frames of each frame image are obtained.

[0063] In some optional implementations, the process of obtaining the edge template of each frame of image includes:

[0064] Perform edge detection on the original image of each frame to obtain edge information of each frame;

[0065] Binarize the edge information of each frame of image to obtain the edge template of each frame of image.

[0066] In some alternative embodiments, the instance segmentation model employs the HTC model.

[0067] In some alternative embodiments, during the training process of the HTC model, the Copy-Paste and Mixup fusion data augmentation methods are used to augment the training set.

[0068] In some alternative embodiments, the device further includes:

[0069] An initialization module, configured to initialize multiple trajectories by using multiple detection boxes of the first frame image of the target video, where each target corresponds to one trajectory;

[0070] A final trajectory module, configured to update the multiple trajectories by using multiple detection boxes of each frame image other than the first frame image of the target video, so as to obtain the final trajectory of each target;

[0071] Wherein, the updating process of the multiple trajectories includes:

[0072] Perform label embedding processing on multiple sub-images corresponding to each frame of image to obtain label information corresponding to each frame of image, and the label information corresponding to each frame of image includes embedded feature information corresponding to multiple detection boxes of each frame of image;

[0073] Use the label information corresponding to the k-th frame image and the label information corresponding to the (k - 1)-th frame image to associate at least one detection box of the k-th frame image with one of the multiple trajectories respectively, so as to update the trajectory associated with at least one detection box, 1 < k ≤ N, N is the number of frames of the target video, and N is an integer greater than 1.

[0074] In some alternative embodiments, the use of the label information corresponding to the k-th frame image and the label information corresponding to the (k - 1)-th frame image to associate at least one detection box of the k-th frame image with one of the multiple trajectories respectively includes:

[0075] Respectively use each detection box of the k-th frame image as the target detection box, and calculate the surface distance between the embedded feature information corresponding to the target detection box and the embedded feature information corresponding to each detection box of the (k - 1)-th frame image one by one;

[0076] When the surface distance between the embedded feature information corresponding to one and only one associated detection frame in the multiple detection frames of the k-1th frame image and the embedded feature information corresponding to the target detection frame is not greater than a preset distance, the trajectory of the target corresponding to the target detection frame and the associated detection frame is associated.

[0077] In some optional implementations, the updating process of the multiple trajectories further includes:

[0078] When the surface distance between the embedded feature information corresponding to any one of the multiple detection frames in the k-1th frame image and the embedded feature information corresponding to the target detection frame is greater than the preset distance, calculating the IOU between the target detection frame and the predicted detection frame of each trajectory in the kth frame image one by one;

[0079] When the IOU between one and only one associated prediction detection box and the target detection box in the prediction detection box of the k-th frame image is not greater than a preset threshold, the target detection box and the trajectory of the target corresponding to the associated prediction detection box are associated.

[0080] In some optional implementations, performing label embedding processing on the multiple sub-images corresponding to each frame of image to obtain label information corresponding to each frame of image includes:

[0081] The MGN model is used to perform label embedding processing on multiple sub-images corresponding to each frame image to obtain the label information corresponding to each frame image;

[0082] The MGN model is obtained by training the preset Resnet50-ibn-a network using the training set.

[0083] In some optional embodiments, the preset Resnet50-ibn-a network is used to perform feature extraction on the image to be processed to obtain global features and local features;

[0084] During the training process of the MGN model, Triplet loss and Softmax loss are used to calculate the loss of the extracted global features, and Softmax loss is used to calculate the loss of the extracted local features.

[0085] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0086] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which implements the steps of any of the above methods when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The present application is further described below with reference to the accompanying drawings and implementation methods.

[0088] Figure 1 The figure shows a flow chart of a video instance segmentation method provided by the present application.

[0089] Figure 2 A schematic diagram of a process for obtaining a detection frame provided by the present application is shown.

[0090] Figure 3 A schematic diagram of a process for obtaining an edge template provided by the present application is shown.

[0091] Figure 4 A schematic diagram of the structure of Mask R-CNN provided in this application is shown.

[0092] Figure 5 A schematic diagram of the structure of a Cascade R-CNN provided in this application is shown.

[0093] Figure 6 A structural diagram of an HTC model provided by this application is shown.

[0094] Figure 7 A flow chart of another video instance segmentation method provided by the present application is shown.

[0095] Figure 8 A structural diagram of an MGN model provided by this application is shown.

[0096] Figure 9 A schematic diagram of the structure of Resnet50 provided in this application is shown.

[0097] Figure 10 A schematic structural diagram of a video instance segmentation device provided by the present application is shown.

[0098] Figure 11 A schematic structural diagram of a detection frame acquisition module provided in this application is shown.

[0099] Figure 12 A schematic structural diagram of another video instance segmentation device provided by the present application is shown.

[0100] Figure 13 The figure shows a structural block diagram of an electronic device provided by the present application.

[0101] Figure 14 FIG. 3 is a schematic structural diagram of a program product 300 for implementing a video instance segmentation method provided by the present application. DETAILED DESCRIPTION

[0102] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0103] This application relates to artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, giving them the capabilities of perception, reasoning, and decision-making.

[0104] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, smart transportation, and automated control.

[0105] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0106] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0107] See also Figure 1 , Figure 1 The following is a flow chart of a method for instance segmentation of a video provided by the present application. The method is used to perform instance segmentation on multiple objects in a video, and the method includes:

[0108] Step S101: Acquire a target video, wherein the target video is obtained by capturing multiple targets in a preset area using a video capture device;

[0109] Step S102: obtaining multiple detection frames for each frame of the target video, wherein each target corresponds to one or more detection frames;

[0110] Step S103: using the multiple detection frames of each frame image, obtaining multiple sub-images corresponding to each frame image, wherein each target corresponds to a sub-image;

[0111] Step S104: segmenting the target in each sub-image corresponding to each frame of image using the instance segmentation model to obtain a segmentation result of each target corresponding to each frame of image.

[0112] This application does not limit the video acquisition device, and the video acquisition device is, for example, a camera, including an optical camera and / or an infrared camera. The preset area is, for example, the shooting area (imaging area) of a (monocular) fixed-position camera.

[0113] The target video includes a temporally continuous sequence of multiple image frames. For example, the target video may include a first image frame, a second image frame, ..., an N-th image frame, where N is the number of frames in the target video and is an integer greater than 1. For example, N is 2, 3, 5, 10, 100, or 1000.

[0114] This application does not limit the number of targets of multiple targets, for example, 2, 3, 5, 10, 100, 1000.

[0115] In one embodiment, the plurality of targets are all moving targets. In another embodiment, a portion of the plurality of targets are moving targets and the remaining portion are stationary targets. In yet another embodiment, the plurality of targets are all stationary targets.

[0116] This application does not limit the target type. The target is, for example, a biological body, and the biological body is, for example, a person (such as a pedestrian) or an animal.

[0117] In one embodiment, the detection frame corresponding to each target may include one or more of a visible range position frame, a full body range position frame, and a head range position frame. Generally speaking, each target corresponds to a unique visible range position frame, a unique full body range position frame, and a unique head range position frame. Since each target corresponds to one or more detection frames and each target corresponds to a sub-image, sub-images and detection frames do not necessarily correspond one-to-one. However, for each target, a sub-image can be obtained using one of the detection frames corresponding to the target.

[0118] The segmentation result of each object is, for example, contour information of each object.

[0119] Thus, a video capture device is used to capture multiple targets in a preset area to obtain a target video. For each frame of the target video, multiple detection frames are set to locate each target in the image. The sub-image corresponding to each target is obtained using these detection frames. The target in each sub-image is segmented using an instance segmentation model to obtain a segmentation result for each target. Since each target corresponds to one or more detection frames, the sub-image corresponding to each target is obtained through the detection frames. When segmenting each target, only the portion of the image corresponding to the single target (i.e., the sub-image) can be segmented. Compared with segmenting each target from the entire image (i.e., the original image in the target video), the amount of computation can be reduced and the efficiency of video instance segmentation can be improved. On the other hand, since the detection frame of each target is first located, the single target is easy to segment. Therefore, each target in the video can be quickly located and segmented in complex scenes. The moving target can be automatically and effectively segmented from the image sequence corresponding to the video in real time, thereby improving the effectiveness and accuracy of video instance segmentation and solving the problem of poor segmentation effect of a single target.

[0120] This application does not limit the method for obtaining the detection frame. In some optional embodiments, it can be to receive given detection frame position information and thereby locate the detection frame. In other optional embodiments, it can be to use an object detection algorithm to directly perform object detection on the original image of each frame, thereby obtaining multiple detection frames for each frame.

[0121] See also Figure 2 , Figure 2 A schematic diagram of a process for obtaining a detection frame provided by the present application is shown. In some further optional implementations, the step S102 may include:

[0122] Step S201: using a frame difference method to obtain multiple target images for each frame;

[0123] Step S202: Based on the multiple target images in each frame image, multiple detection frames are obtained for each frame image.

[0124] The frame difference method (also known as the inter-frame difference method) is based on the principle that when a moving object is present in a video, there will be a difference in grayscale between adjacent frames (or three adjacent frames). By calculating the absolute value of the grayscale difference between the two frames, a stationary object will appear entirely zero in the difference image. However, a moving object, particularly its outline, will show non-zero grayscale variations. When the absolute value exceeds a certain threshold, it is identified as a moving target, thus enabling target detection. The advantages of the frame difference method include simple algorithm implementation and low programming complexity. It is also relatively insensitive to scene changes such as lighting, adapts to various dynamic environments, and exhibits strong robustness. However, its disadvantages are that it cannot extract the entire object area; instead, it can only extract the boundary, which is coarse and often larger than the actual object. Fast-moving objects can easily cause ghosting, or even be detected as two separate moving objects. For slow-moving objects, if the objects in the previous and next frames almost completely overlap, they cannot be detected. Therefore, this method is generally suitable for simple real-time motion detection. The frame difference method may include, for example, a two-frame method (adjacent frame difference method) and a three-frame method (three-frame difference method).

[0125] Therefore, the frame difference method (separation detection) is first used to obtain multiple target images in each frame, and then a detection frame is set for the target image in each frame, obtaining multiple detection frame information for a single frame. On the one hand, first obtaining the target image in each frame and then using the target image to set the detection frame means that the detection frame setting is divided into two steps. A target image is added as intermediate information between each frame image (i.e., the original image) and the detection frame (position information). The target image is a part of the original image. Compared with setting the detection frame using the original image, setting the detection frame using the target image reduces the computational complexity. On the other hand, the image sequence corresponding to the video captured by the video acquisition device is continuous. If there is no moving target in the scene, the change between consecutive frames is very weak. If there is a moving target, there will be obvious change between consecutive frames. Using the frame difference method to obtain the target image and analyze the spatial motion characteristics of the target and the correlation between multiple targets can effectively utilize the spatial motion characteristics of the target and the inherent relationship between multiple targets, solving the problem of difficult detection of moving targets due to mutual occlusion or irregular target motion.

[0126] In some optional implementations, multiple target images of each frame (obtained by the frame difference method) may be used to directly obtain multiple detection frames of each frame.

[0127] In some other optional implementations, multiple target images of each frame (obtained by the frame difference method) may be updated, and the detection frame may be obtained using the updated target images. The step S202 may include:

[0128] The target image is updated using the edge template of each frame image to remove holes and ghosting of the target image contour;

[0129] Based on the updated multiple target images of each frame image, multiple detection frames of each frame image are obtained.

[0130] The frame difference method is used to quickly obtain multiple target images. However, the contours of the target images obtained at this time are prone to "holes" and "ghosting", making it difficult to ensure their integrity, resulting in inaccurate target detection. Therefore, the target image cannot be used directly as an object template. Instead, it is necessary to combine the (spatial) edge detection algorithm to determine the exact edge of each target in the original image through edge detection to obtain the edge template in the image. The edge template is then used to update the incomplete target image obtained by the frame difference method to remove holes and ghosting in the contours of the target image, thereby making up for the shortcomings of a single algorithm (i.e., the frame difference method), thereby segmenting a complete and accurate moving target and solving the problem of inaccurate target segmentation boundaries.

[0131] The present application does not limit the method of obtaining the edge template. In some optional implementations, it may be to receive a given edge template.

[0132] See also Figure 3 , Figure 3 A schematic diagram of a process for obtaining an edge template provided by the present application is shown. In other optional embodiments, the process of obtaining an edge template for each frame of image may include:

[0133] Step S301: performing edge detection on the original image of each frame to obtain edge information of each frame;

[0134] Step S302: binarize edge information of each frame of image to obtain an edge template of each frame of image.

[0135] In one embodiment, an edge detection algorithm is used to perform edge detection on the original image of each frame. The edge detection algorithm can use the discontinuous nature of grayscale values ​​to segment the target area based on grayscale mutations (obtaining the distribution range of the target area). This application does not limit the edge detection algorithm used, which can be the image edge detection method disclosed in patent CN104077773A, the image edge detection method disclosed in patent CN106097306B, or the image edge detection method disclosed in patent CN108629788A.

[0136] Therefore: use the (spatial) edge detection algorithm to perform edge detection on the original image of each frame of image, obtain the edge information (of the target area) of the original image as the edge information of each frame of image, and then binarize the obtained edge information to obtain an edge template for each frame of image. Compared with other data processing methods, binarization has a small amount of calculation, high calculation efficiency, and short calculation time, which can improve the efficiency of the overall video segmentation method.

[0137] This application does not limit the instance segmentation model. In some optional implementations, the instance segmentation model may use Mask R-CNN.

[0138] See also Figure 4 , Figure 4 The following figure shows a schematic diagram of the structure of a Mask R-CNN provided by this application. Mask-RCNN can complete various tasks such as object classification, object detection, semantic segmentation, instance segmentation, and human pose estimation by adding different branches. When Mask-RCNN is used for instance segmentation, it adds a branch for semantic segmentation on the basis of Faster-RCNN (classification + regression branch). First, input an image to be processed, and then perform the corresponding preprocessing operations, or directly input the preprocessed image; then, input it into a pre-trained neural network (ResNeXt, etc.) to obtain the corresponding feature map; then, set a predetermined number of ROIs (Region of Interest, image region of interest) for each point in this feature map to obtain multiple candidate ROIs; then, send these candidate ROIs to the RPN network (Region Proposal Network) for binary classification (foreground or background) and BB regression to filter out some candidate ROIs; then, perform ROIAlign operation on these remaining ROIs (that is, first match the pixels of the original image and the feature map, and then match the feature map with the fixed feature); finally, classify these ROIs (multi-category classification), BB regression (Bounding-box regression) and generate MASK (perform FCN operation in each ROI).

[0139] In other optional implementations, the instance segmentation model can be Instance-sensitive FCN, FCIS, YOLCAT, PolarMask, SOLO, RDSNet & PointRend, or BlendMask. The above models are all prior art and are not described in detail in this application. For details, please refer to "The Latest and Most Comprehensive Review of Instance Segmentation: From Mask R-CNN to BlendMask" (online link address: https: / / cloud.tencent.com / developer / article / 1594081).

[0140] In some further optional implementations, the instance segmentation model may adopt an HTC model.

[0141] See also Figure 5 , Figure 5 The figure shows a schematic diagram of the structure of a Cascade R-CNN provided by the present application. CascadeRCNN consists of a series of detectors with different IOU thresholds to provide more choices for detection boxes close to false positives. The entire network is trained stage by stage, and the network of the current stage provides a better distribution for the network training of the next stage. The resampling of the gradually improved hypothesis ensures that all detectors have a set of positive examples of equivalent size, thereby reducing the overfitting problem. Cascade RCNN is a cascade regression problem structure with a total of T cascade stages, each of which is optimal for its input b. In this way, the quality of Bbox is gradually improved.

[0142] See also Figure 6 , Figure 6 The figure shows a schematic diagram of the structure of an HTC model provided by this application. The HTC (Hybrid Task Cascade) model integrates the Cascade R-CNN and Mask R-CNN models, changing the box branches from parallel to series (i.e., the box branch is executed first in each stage, and the regressed detection box is then handed over to the mask branch to predict the mask). Information flow is also introduced between mask branches (similar to the information flow between box branches in Cascade R-CNN). By introducing the semantic segmentation branch S to provide spatial context information, the HTC model can distinguish the foreground (i.e., the target image) and background (the part outside the target image) of each frame.

[0143] The HTC model directly borrows from Cascade R-CNN, connecting Mask R-CNNs from adjacent stages through box branches, which significantly improves box AP. The parallel branch structure in Mask R-CNN is replaced with a serial branch structure, providing more accurate ROI information for the mask branch. Mask features from different stages are no longer isolated, allowing for information transfer. Backpropagation during training ensures that all mask features are supervised, resulting in a significant improvement in mask AP (Average Precision).

[0144] The present application does not limit the data enhancement method. In some optional implementations, during the training process of the HTC model, a Mosaic data enhancement method is used to enhance the training set data.

[0145] In other optional implementations, during the training process of the HTC model, the training set is enhanced using a data enhancement method that combines Copy-Paste and Mixup.

[0146] Copy-Paste involves mixing the contents of two images using a preset formula. This process is highly random, making it a robust data augmentation method with a simple strategy that can be easily integrated into any instance segmentation codebase without increasing training costs or inference time. Mixup is an algorithm used in computer vision to perform mixed-class augmentation on images. It can mix images of different classes to expand the training set, making the trained model more stable in terms of model predictions and gradient modulus between training images. A data augmentation method that fuses the two algorithms mentioned above can, to a certain extent, combine the advantages of both.

[0147] Specifically, for a single image, objects can be copied from multiple other images and randomly pasted. The pasted portions are then fused using a Mixup strategy. This avoids occluding large portions of the object due to direct pasting and eliminates the need to modify labels, making the entire process much simpler. This data augmentation method, combining Copy-Paste and Mixup, addresses object occlusion and increases the robustness of the model.

[0148] This application does not limit the target tracking method. In some optional implementations, the target tracking method disclosed in patent CN112489077A, or the object tracking method disclosed in patent CN112712051A, or the target tracking method disclosed in patent CN113674318A can be adopted.

[0149] See also Figure 7 , Figure 7 shows a schematic flowchart of another video instance segmentation method provided by this application. In some other optional embodiments, the method may further include:

[0150] Step S105: Initialize multiple trajectories using multiple detection boxes of the first frame image of the target video, where each target corresponds to one trajectory;

[0151] Step S106: For each frame image other than the first frame image of the target video, use multiple detection boxes of each frame image to update the multiple trajectories to obtain the final trajectory of each target;

[0152] Among them, the update process of the multiple trajectories includes:

[0153] Perform label embedding processing on multiple sub-images corresponding to each frame image to obtain label information corresponding to each frame image, and the label information corresponding to each frame image includes embedding feature information corresponding to multiple detection boxes of each frame image;

[0154] Use the label information corresponding to the k-th frame image and the label information corresponding to the (k - 1)-th frame image to associate at least one detection box of the k-th frame image with one of the multiple trajectories respectively, so as to update the trajectory associated with at least one detection box, 1 < k ≤ N, where N is the number of frames of the target video, and N is an integer greater than 1.

[0155] This application does not limit N. When N = 1, k = 2. Use the label information corresponding to the second frame image and the label information corresponding to the first frame image to associate at least one detection box of the second frame image with one of the multiple trajectories respectively, so as to update the trajectory associated with at least one detection box. For example: Before the update, the trajectory associated with a detection box includes 1 detection box position; after the update, the trajectory associated with this detection box includes 2 detection box positions.

[0156] When N=5, k is 2, 3, 4, and 5, respectively. Using the label information corresponding to the second image frame and the label information corresponding to the first image frame, at least one detection frame of the second image frame is associated with one of the multiple trajectories, thereby updating the trajectory associated with the at least one detection frame. Using the label information corresponding to the third image frame and the label information corresponding to the second image frame, at least one detection frame of the third image frame is associated with one of the multiple trajectories, thereby updating the trajectory associated with the at least one detection frame. Using the label information corresponding to the fourth image frame and the label information corresponding to the fourth image frame, at least one detection frame of the fourth image frame is associated with one of the multiple trajectories, thereby updating the trajectory associated with the at least one detection frame. Using the label information corresponding to the fifth image frame and the label information corresponding to the fourth image frame, at least one detection frame of the fifth image frame is associated with one of the multiple trajectories, thereby updating the trajectory associated with the at least one detection frame. After the trajectories are associated and updated using the fifth image frame, the trajectory associated with the target includes five detection frame positions. For example, if a detection box matches the associated track in the second, third, fourth, and fifth frames, the associated track is updated four times. After the fourth update, the track associated with the detection box includes five detection box positions.

[0157] In one embodiment, the trajectory corresponding to the target is a sequence of detection frame positions of the target. For example, if the target Zhang San appears in the first frame to the 58th frame of the target video, the trajectory corresponding to the target Zhang San may include: the detection frame position of the first frame, the detection frame position of the second frame, ..., the detection frame position of the 58th frame. For another example, if the target Li Si appears in the 10th frame to the 106th frame of the target video, the trajectory corresponding to the target Li Si may include: the detection frame position of the 10th frame, the detection frame position of the 11th frame, ..., the detection frame position of the 106th frame. When the detection frame corresponding to the target Zhang San appears in an intermediate frame X (from the first frame to the 58th frame), the trajectory corresponding to the target Zhang San is activated in the intermediate frame X; when the detection frame corresponding to the target Zhang San does not appear in an intermediate frame Y, the trajectory corresponding to the target Zhang San is deactivated in the intermediate frame Y.

[0158] Label embedding refers to converting data into appropriate feature representations. Appropriate means that the specific attributes of the sample can be expressed using the concept of distance. Through embedding, the original sample data can be converted into feature data suitable for machine learning. For example, for the same set of speech samples, a speaker-distinguishing model can convert the speech sample into a numerical vector, and the embedding features (i.e., embedded feature information) corresponding to the speech samples from the same person have a smaller Euclidean distance; however, for another model that recognizes the content of speech, the embedding features corresponding to the speech that speaks the same words (rather than the speech of the same person) will have a smaller Euclidean distance. Embedding features are features that make it convenient to use distance to express the degree of similarity of attributes. The distance between features corresponding to samples with similar attributes is smaller.

[0159] Moreover, embedding can learn a low-dimensional (about 100 dimensions) vector for each pixel that can effectively represent the pixel, and ultimately use the vector or the variable score generated by the vector as the input variable for other machine learning tasks.

[0160] In this way, label embedding is used to track the trajectory of each target. First, label embedding is performed on multiple sub-images of each frame. Since each sub-image corresponds to one or more detection frames, embedded feature information corresponding to each detection frame is obtained. The embedded feature information can be used to calculate the feature distance between adjacent frames. In the process of obtaining the final trajectory, multiple trajectories are first initialized based on the multiple detection frames in the first frame image. In subsequent frames, for each detection frame in the current frame image, an attempt is first made to associate the detection frame with the existing trajectory based on the feature distance between adjacent frames, thereby updating the associated trajectory (i.e., updating the detection frame position sequence in the trajectory). Analyzing the embedded feature information of the target in adjacent frames can make good use of the target's time domain characteristics, and the resulting associated trajectory is more reliable.

[0161] This application does not limit the method for associating the detection box and the trajectory. In some optional implementations, the detection box and the trajectory can be associated entirely based on spatial distance IOU. IOU (Intersection over Union) is the full name of the intersection over union ratio. IOU calculates the ratio of the intersection and union of the "predicted detection box" and the "true detection box".

[0162] In some other optional implementations, the associating at least one detection frame of the k-th image frame with one of the multiple trajectories using the label information corresponding to the k-th image frame and the label information corresponding to the k-1-th image frame may include:

[0163] Taking each detection frame of the k-th frame image as a target detection frame, respectively, calculating the surface distance between the embedded feature information corresponding to the target detection frame and the embedded feature information corresponding to each detection frame of the k-1-th frame image;

[0164] When the surface distance between the embedded feature information corresponding to one and only one associated detection frame in the multiple detection frames of the k-1th frame image and the embedded feature information corresponding to the target detection frame is not greater than a preset distance, the trajectory of the target corresponding to the target detection frame and the associated detection frame is associated.

[0165] The preset distance is a pre-set distance, such as 0.6, 0.7, or 0.8.

[0166] The surface distance between the embedded feature information corresponding to the target detection frame and the embedded feature information corresponding to each detection frame of the k-1th frame image, for example, refers to the average surface distance between the target surface point set corresponding to the target detection frame and the target surface point set corresponding to each detection frame of the k-1th frame image.

[0167] When associating a target detection frame with multiple existing tracks, the surface distance between adjacent frames is first used for association. If, among the multiple surface distances between the target detection frame and multiple detection frames in the previous frame image, there is only one associated detection frame whose surface distance with the target detection frame is within a preset distance range, the target detection frame is associated with the target track corresponding to the unique associated detection frame. Detection frames are typically rectangular and contain both target and non-target areas. For detection frames with a large proportion of non-target areas (such as a giraffe with a crooked neck), there may be cases where the Z position of the detection frame in the previous frame is close to that of the target detection frame, but they are not the same target. Using the characteristic distance of the target surface of adjacent frames for association is more accurate and reliable than associating based solely on the position information of the detection frames.

[0168] In addition to using surface (feature) distance to associate the detection box with the existing track, the spatial distance can also be used to associate the detection box with the existing track. In some optional embodiments, the updating process of the multiple tracks can also include:

[0169] When the surface distance between the embedded feature information corresponding to any one of the multiple detection frames in the k-1th frame image and the embedded feature information corresponding to the target detection frame is greater than the preset distance, calculating the IOU between the target detection frame and the predicted detection frame of each trajectory in the kth frame image one by one;

[0170] When the IOU between one and only one associated prediction detection box and the target detection box in the prediction detection box of the k-th frame image is not greater than a preset threshold, the target detection box and the trajectory of the target corresponding to the associated prediction detection box are associated.

[0171] The preset threshold is a pre-set threshold, such as 0.7, 0.8, or 0.9.

[0172] Therefore, when the surface distance association of the detection frames between adjacent frames fails, the IOU between the target detection frame and the predicted detection frame of each trajectory in the k-th frame image is used for association. The IOU calculation is simple, has a small amount of calculation, a fast calculation speed, and a short calculation time, and can quickly achieve association.

[0173] In other words, surface distance can be used to associate the target detection box with the existing track. When some detection boxes cannot be successfully associated with the existing track, spatial distance (IOU algorithm) can be used to associate these detection boxes with the track. Among them, for both activated tracks and inactivated tracks (but not completely lost), the IOU algorithm can be used to calculate the spatial distance between them and the (current) target detection box, and then associate the target detection box with the existing track.

[0174] In this way, the surface distance of the detection frame is mainly used, supplemented by the spatial distance of the detection frame. The relationship between the previous and next frames of the image in the video is utilized to explore the spatial motion characteristics of the target and the connection between multiple targets, thereby improving the accuracy of specific target separation in complex scenes.

[0175] This application does not limit the label embedding method. In some optional implementations, it can be possible to embed labels into the original image of each frame.

[0176] See also Figure 8 and Figure 9 , Figure 8 shows a structural diagram of an MGN model provided by this application, Figure 9 A schematic diagram of the structure of Resnet50 provided in this application is shown. In other optional implementations, the label embedding process for the multiple sub-images corresponding to each frame of image to obtain the label information corresponding to each frame of image may include:

[0177] The MGN model is used to perform label embedding processing on multiple sub-images corresponding to each frame image to obtain the label information corresponding to each frame image;

[0178] The MGN model is obtained by training the preset Resnet50-ibn-a network using the training set.

[0179] This application does not limit the pre-training model of the MGN model. In other optional implementations, it can be Resnet101.

[0180] This application does not limit the training process of the MGN model. In some optional implementations, common training techniques in ReID can be used to optimize the performance of the MGN model.

[0181] The MGN model is trained using the preset Resnet50-ibn-a network, and then used to perform label embedding on sub-images. During forward propagation, shallow-layer features can be reused in deeper layers. During backward propagation, gradients from deeper layers can be directly passed back to shallow layers. Only the residual error needs to be propagated during error backpropagation, thus solving the vanishing gradient (network degradation) problem. An MGN (multiple granularity network) combines global features with multi-granularity local features. Global features are responsible for extracting shared features at the macro level. The image is then divided into different blocks, each with different granularity, responsible for extracting features at different levels or layers. Combining global and local features allows for rich information and details to fully represent the input image. Practice has shown that as the segmentation granularity increases, the model learns more detailed information. The MGN model is used to embed labels for multiple sub-images in each frame image to obtain the embedded feature information corresponding to each sub-image (a combination of global features and local features). Since the sub-image corresponds to one or more detection frames of the current frame image, the embedded feature information corresponding to each detection frame is obtained.

[0182] In some optional embodiments, the preset Resnet50-ibn-a network is used to perform feature extraction on the image to be processed to obtain global features and local features;

[0183] During the training process of the MGN model, Triplet loss and Softmax loss are used to calculate the loss of the extracted global features, and Softmax loss is used to calculate the loss of the extracted local features.

[0184] Global features are complete but rough, while local features are incomplete but fine. Using Triplet loss and Softmaxloss to calculate the loss of the extracted global features can take into account the globality and accuracy of the training process; using Softmaxloss to calculate the loss of the extracted local features can learn local fine information and avoid the situation where local information is ignored.

[0185] In other optional implementations, only Softmax loss may be used to calculate the loss of the extracted global features and local features.

[0186] In some optional implementations, Triplet loss and Softmax loss may be used simultaneously to calculate the losses of the extracted global features and local features.

[0187] See also Figure 10 , Figure 10 The schematic diagram of the structure of a video instance segmentation device provided by the present application is shown. The specific implementation method is consistent with the implementation method and the technical effect achieved in the above-mentioned method implementation method, and some contents are not repeated here. The (video instance segmentation) device includes:

[0188] The video acquisition module 101 is used to acquire a target video, wherein the target video is obtained by capturing multiple targets in a preset area using a video acquisition device;

[0189] A detection frame acquisition module 102 is configured to acquire multiple detection frames for each frame of the target video, wherein each target corresponds to one or more detection frames;

[0190] The sub-image acquisition module 103 is used to obtain multiple sub-images corresponding to each frame of the image using multiple detection frames of each frame of the image, wherein each target corresponds to a sub-image;

[0191] The object segmentation module 104 is configured to segment the object in each sub-image corresponding to each frame of image using an instance segmentation model to obtain a segmentation result of each object corresponding to each frame of image.

[0192] See also Figure 11 , Figure 11 FIG2 shows a schematic diagram of the structure of a detection frame acquisition module 102 provided in the present application. In some optional implementations, the detection frame acquisition module 102 may include:

[0193] The target image unit 201 is used to obtain multiple target images of each frame image by using a frame difference method;

[0194] The detection frame unit 202 is configured to obtain multiple detection frames for each frame of image based on multiple target images in each frame of image.

[0195] In some optional implementations, the detection frame unit 202 may be used to:

[0196] The target image is updated using the edge template of each frame image to remove holes and ghosting of the target image contour;

[0197] Based on multiple target images of each updated frame image, obtain multiple detection boxes for each frame image.

[0198] In some optional embodiments, the process of obtaining the edge template of each frame image includes:

[0199] Perform edge detection on the original image of each frame image to obtain the edge information of each frame image;

[0200] Perform binarization on the edge information of each frame image to obtain the edge template of each frame image.

[0201] In some optional embodiments, the instance segmentation model uses the HTC model.

[0202] In some optional embodiments, during the training process of the HTC model, the Copy-Paste and Mixup fusion data augmentation methods are used to perform data augmentation on the training set.

[0203] See Figure 12 , Figure 12 shows a schematic structural diagram of another video instance segmentation device provided by the present application. In some optional embodiments, the device may further include:

[0204] Initialization module 105, configured to initialize multiple trajectories by using multiple detection boxes of the first frame image of the target video, where each target corresponds to one trajectory;

[0205] Final trajectory module 106, configured to, for each frame image other than the first frame image of the target video, update the multiple trajectories by using multiple detection boxes of each frame image to obtain the final trajectory of each target;

[0206] Among them, the update process of the multiple trajectories includes:

[0207] Perform label embedding processing on multiple sub-images corresponding to each frame image to obtain label information corresponding to each frame image, and the label information corresponding to each frame image includes embedding feature information corresponding to multiple detection boxes of each frame image;

[0208] Utilize the label information corresponding to the k-th frame image and the label information corresponding to the (k - 1)-th frame image, and associate at least one detection box of the k-th frame image with one of the multiple trajectories respectively to update the trajectory associated with at least one detection box, 1 < k ≤ N, N is the number of frames of the target video, and N is an integer greater than 1.

[0209] In some optional implementations, associating at least one detection frame of the k-th image frame with one of the multiple trajectories using the label information corresponding to the k-th image frame and the label information corresponding to the k-1-th image frame, respectively, includes:

[0210] Taking each detection frame of the k-th frame image as a target detection frame, respectively, calculating the surface distance between the embedded feature information corresponding to the target detection frame and the embedded feature information corresponding to each detection frame of the k-1-th frame image;

[0211] When the surface distance between the embedded feature information corresponding to one and only one associated detection frame in the multiple detection frames of the k-1th frame image and the embedded feature information corresponding to the target detection frame is not greater than a preset distance, the trajectory of the target corresponding to the target detection frame and the associated detection frame is associated.

[0212] In some optional implementations, the updating process of the multiple trajectories further includes:

[0213] When the surface distance between the embedded feature information corresponding to any one of the multiple detection frames in the k-1th frame image and the embedded feature information corresponding to the target detection frame is greater than the preset distance, calculating the IOU between the target detection frame and the predicted detection frame of each trajectory in the kth frame image one by one;

[0214] When the IOU between one and only one associated prediction detection box and the target detection box in the prediction detection box of the k-th frame image is not greater than a preset threshold, the target detection box and the trajectory of the target corresponding to the associated prediction detection box are associated.

[0215] In some optional implementations, performing label embedding processing on the multiple sub-images corresponding to each frame of image to obtain label information corresponding to each frame of image includes:

[0216] The MGN model is used to perform label embedding processing on multiple sub-images corresponding to each frame image to obtain the label information corresponding to each frame image;

[0217] The MGN model is obtained by training the preset Resnet50-ibn-a network using the training set.

[0218] In some optional embodiments, the preset Resnet50-ibn-a network is used to perform feature extraction on the image to be processed to obtain global features and local features;

[0219] During the training process of the MGN model, Triplet loss and Softmax loss are used to calculate the loss of the extracted global features, and Softmax loss is used to calculate the loss of the extracted local features.

[0220] See also Figure 13 , Figure 13 FIG2 shows a block diagram of an electronic device 200 provided by the present application. The electronic device 200 includes at least one memory 210, at least one processor 220, and a bus 230 for connecting different platform systems.

[0221] The memory 210 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 211 and / or a cache memory 212 , and may further include a read-only memory (ROM) 213 .

[0222] Among them, the memory 210 also stores a computer program, which can be executed by the processor 220, so that the processor 220 implements the steps of any of the above methods. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the above method implementation method, and some contents will not be repeated here.

[0223] The memory 210 may also include a utility 214 having at least one program module 215, such program module 215 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0224] Accordingly, the processor 220 may execute the aforementioned computer program and the utility 214 .

[0225] Bus 230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0226] The electronic device 200 may also communicate with one or more external devices 240, such as a keyboard, pointing device, Bluetooth device, etc., and may also communicate with one or more devices capable of interacting with the electronic device 200, and / or any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may be performed via an input / output interface 250. Furthermore, the electronic device 200 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0227] The present application also provides a computer-readable storage medium for storing a computer program. When the computer program is executed, the steps of any of the above methods are implemented. The specific implementation method is consistent with the implementation method and the technical effect achieved in the implementation method of the above method, and some contents are not repeated here.

[0228] See also Figure 14 , Figure 14 The present application provides a schematic diagram of a program product 300 for implementing a method for video instance segmentation. Program product 300 may be implemented in a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, program product 300 of the present invention is not limited thereto. In the present application, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Program product 300 may utilize any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0229] A computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, or any suitable combination thereof. The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0230] This application is explained from the perspectives of purpose of use, effectiveness, progress and novelty, and has complied with the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings of this application are only preferred embodiments of this application and are not intended to limit this application. Therefore, all structures, devices, features, etc. that are similar or identical to those of this application, that is, all equivalent replacements or modifications made in accordance with the scope of the patent application of this application, should fall within the scope of protection of the patent application of this application.

Claims

1. A video instance segmentation method, characterized in that: The method includes: Obtaining a target video, which is obtained by using a video acquisition device to collect multiple targets in a preset area; Obtaining multiple detection boxes for each frame image of the target video, where each target corresponds to one or more detection boxes; Using the multiple detection boxes of each frame image to obtain multiple sub-images corresponding to each frame image, where each target corresponds to one sub-image; Using an instance segmentation model to segment the targets in each sub-image corresponding to each frame image to obtain the segmentation result of each target corresponding to each frame image; The method further includes: Initializing multiple trajectories by using the multiple detection boxes of the first frame image of the target video, where each target corresponds to one trajectory respectively; For each frame image other than the first frame image of the target video, using the multiple detection boxes of each frame image to update the multiple trajectories to obtain the final trajectory of each target; Wherein, the update process of the multiple trajectories includes: Performing label embedding processing on the multiple sub-images corresponding to each frame image to obtain the label information corresponding to each frame image, and the label information corresponding to each frame image includes the embedding feature information corresponding to the multiple detection boxes of each frame image; Using the label information corresponding to the k-th frame image and the label information corresponding to the (k - 1)-th frame image, associating at least one detection box of the k-th frame image with one of the multiple trajectories respectively to update the trajectory associated with at least one detection box, 1 < k ≤ N, N is the number of frames of the target video, and N is an integer greater than 1.

2. The video instance segmentation method according to claim 1, characterized in that The obtaining of the multiple detection boxes for each frame image of the target video includes: Adopting the frame difference method to obtain multiple target images for each frame image; Based on the multiple target images of each frame image, obtaining the multiple detection boxes for each frame image.

3. The video instance segmentation method according to claim 2, characterized in that The obtaining of the multiple detection boxes for each frame image based on the multiple target images of each frame image includes: Using the edge template of each frame image to update the target image to remove the holes and ghosts in the contour of the target image; Based on the multiple target images of each frame image after update, obtaining the multiple detection boxes for each frame image.

4. The video instance segmentation method according to claim 3, wherein: The obtaining process of the edge template of each frame image includes: Performing edge detection on the original image of each frame image to obtain the edge information of each frame image; Performing binarization on the edge information of each frame image to obtain the edge template of each frame image.

5. The video instance segmentation method according to claim 1, wherein: The instance segmentation model adopts the HTC model.

6. The video instance segmentation method according to claim 5, characterized in that During the training process of the HTC model, the Copy-Paste and Mixup fusion data augmentation methods are used to perform data augmentation on the training set.

7. The video instance segmentation method according to claim 1, characterized in that The using of the label information corresponding to the k-th frame image and the label information corresponding to the (k - 1)-th frame image to associate at least one detection box of the k-th frame image with one of the multiple trajectories respectively includes: Respectively taking each detection box of the k-th frame image as a target detection box, and calculating the surface distance between the embedding feature information corresponding to the target detection box and the embedding feature information corresponding to each detection box of the (k - 1)-th frame image one by one; When the surface distance between the embedded feature information corresponding to one and only one associated detection frame in the multiple detection frames of the k-1th frame image and the embedded feature information corresponding to the target detection frame is not greater than a preset distance, the trajectory of the target corresponding to the target detection frame and the associated detection frame is associated.

8. The video instance segmentation method according to claim 7, characterized in that: The updating process of the plurality of trajectories further includes: When the surface distance between the embedded feature information corresponding to any one of the multiple detection frames in the k-1th frame image and the embedded feature information corresponding to the target detection frame is greater than the preset distance, calculating the IOU between the target detection frame and the predicted detection frame of each trajectory in the kth frame image one by one; When the IOU between one and only one associated prediction detection box and the target detection box in the prediction detection box of the k-th frame image is not greater than a preset threshold, the target detection box and the trajectory of the target corresponding to the associated prediction detection box are associated.

9. The video instance segmentation method according to claim 1, wherein: The performing label embedding processing on the multiple sub-images corresponding to each frame of image to obtain label information corresponding to each frame of image includes: The MGN model is used to perform label embedding processing on multiple sub-images corresponding to each frame image to obtain the label information corresponding to each frame image; The MGN model is obtained by training the preset Resnet50-ibn-a network using the training set.

10. The video instance segmentation method according to claim 9, characterized in that: The preset Resnet50-ibn-a network is used to extract features of the image to be processed to obtain global features and local features; During the training process of the MGN model, Triplet loss and Softmax loss are used to calculate the loss of the extracted global features, and Softmax loss is used to calculate the loss of the extracted local features.

11. A video instance segmentation device, characterized in that: The device comprises: A video acquisition module is used to acquire a target video, wherein the target video is obtained by capturing multiple targets in a preset area using a video acquisition device; A detection frame acquisition module is used to obtain multiple detection frames for each frame of the target video, wherein each target corresponds to one or more detection frames; A sub-image acquisition module is used to obtain multiple sub-images corresponding to each frame of the image using multiple detection frames of each frame of the image, where each target corresponds to a sub-image; The target segmentation module is used to segment the target in each sub-image corresponding to each frame image using the instance segmentation model to obtain the segmentation result of each target corresponding to each frame image; The apparatus may further comprise: An initialization module, configured to initialize multiple trajectories using multiple detection frames of the first frame image of the target video, wherein each target corresponds to one trajectory; A final trajectory module is configured to update the multiple trajectories using multiple detection frames of each frame image except the first frame image of the target video to obtain a final trajectory of each target; The updating process of the multiple trajectories includes: Performing label embedding processing on multiple sub-images corresponding to each frame of image to obtain label information corresponding to each frame of image, wherein the label information corresponding to each frame of image includes embedded feature information corresponding to multiple detection frames of each frame of image; Using the label information corresponding to the k-th frame image and the label information corresponding to the (k - 1)-th frame image, associate at least one detection box of the k-th frame image with one of the multiple trajectories respectively to update the trajectories associated with the at least one detection box, where 1 < k ≤ N, N is the number of frames of the target video, and N is an integer greater than 1.

12. An electronic device, characterized in that: The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the video instance segmentation method according to any one of claims 1-10 are implemented.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video instance segmentation method according to any one of claims 1-10 are implemented.

Images