Video instance segmentation method, instance segmentation model training method, device and medium

Abstract

A video instance segmentation method, an instance segmentation model training method, a device and a medium are disclosed. The video instance segmentation method comprises: performing feature extraction on a current frame in a to-be-processed video to obtain a current feature map of the current frame; determining a reference instance feature sequence based on a historical instance feature sequence corresponding to each of at least one historical frame located before the current frame in the to-be-processed video; determining a current instance feature sequence of the current frame based on the current feature map, the reference instance feature sequence, and historical instance candidate box information sequence corresponding to a predetermined frame in the at least one historical frame; the historical instance candidate box information sequence comprises candidate box information corresponding to each of a plurality of historical instance candidate boxes in a historical feature map of the predetermined frame; and generating a video instance segmentation result of the current frame via an instance segmentation model based on the current feature map and the current instance feature sequence. The disclosure can reduce the complexity of the video instance segmentation algorithm and improve the video instance segmentation speed.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to computer vision technology, and in particular to a video instance segmentation method, an instance segmentation model training method, an apparatus and a medium. BACKGROUND

[0002] Video instance segmentation is an important problem in the field of computer vision. Current video instance segmentation schemes usually have two kinds: one is to first obtain instance predictions of each frame through a picture instance segmentation model, and then perform cross-frame instance association through an additional multi-target tracking model; the other is to first divide a video into multiple overlapping segments to obtain instance predictions of each segment, and then associate these segments through an instance matching algorithm, wherein each segment can include continuous multiple frames in the video, and any two adjacent segments overlap at least one frame. SUMMARY

[0003] To solve the technical problems of current instance video segmentation schemes, such as the need to perform an explicit association step, high algorithm complexity, and slow video instance segmentation speed, the present disclosure is proposed. Embodiments of the present disclosure provide a video instance segmentation method, an instance segmentation model training method, an apparatus and a medium.

[0004] According to an aspect of an embodiment of the present disclosure, a video instance segmentation method is provided, comprising:

[0005] performing feature extraction on a current frame in a to-be-processed video to obtain a current feature map of the current frame;

[0006] determining a reference instance feature sequence based on respective historical instance feature sequences of at least one historical frame located before the current frame in the to-be-processed video;

[0007] determining a current instance feature sequence of the current frame based on the current feature map, the reference instance feature sequence, and a historical instance candidate box information sequence corresponding to a predetermined frame in the at least one historical frame, wherein the historical instance candidate box information sequence comprises respective candidate box information of a plurality of historical instance candidate boxes in a historical feature map of the predetermined frame;

[0008] generating a video instance segmentation result of the current frame via an instance segmentation model based on the current feature map and the current instance feature sequence.

[0009] According to another aspect of an embodiment of the present disclosure, an instance segmentation model training method is provided, comprising:

[0010] performing feature extraction on a current training frame in a training video to obtain a current training feature map of the current training frame;

[0011] determine a reference training instance feature sequence based on respective historical training instance feature sequences of at least one historical training frame before the current training frame in the training video;

[0012] determine a current training instance feature sequence of the current training frame based on the current training feature map, the reference training instance feature sequence, and a historical training instance candidate box information sequence corresponding to a predetermined training frame in the at least one historical training frame, wherein the historical training instance candidate box information sequence comprises candidate box information corresponding to each of a plurality of historical instance candidate boxes in a historical training feature map of the predetermined training frame;

[0013] generate a video instance segmentation result of the current training frame via an initial instance segmentation model based on the current training feature map and the current training instance feature sequence;

[0014] train the initial instance segmentation model based on the video instance segmentation result of the current training frame, the annotation information of the current training frame, and the annotation information of the predetermined training frame;

[0015] determine the trained initial instance segmentation model as an instance segmentation model in response to the trained initial instance segmentation model meeting a preset training end condition.

[0016] According to still another aspect of the embodiments of the present disclosure, a video instance segmentation device is provided, comprising:

[0017] a first feature extraction module configured to perform feature extraction on a current frame in a to-be-processed video to obtain a current feature map of the current frame;

[0018] a first determination module configured to determine a reference instance feature sequence based on respective historical instance feature sequences of at least one historical frame before the current frame in the to-be-processed video;

[0019] a second determination module configured to determine a current instance feature sequence of the current frame based on the current feature map obtained by the first feature extraction module, the reference instance feature sequence determined by the first determination module, and a historical instance candidate box information sequence corresponding to a predetermined frame in the at least one historical frame, wherein the historical instance candidate box information sequence comprises candidate box information corresponding to each of a plurality of historical instance candidate boxes in a historical feature map of the predetermined frame;

[0020] a first generation module configured to generate a video instance segmentation result of the current frame via an instance segmentation model based on the current feature map obtained by the first feature extraction module and the current instance feature sequence determined by the second determination module.

[0021] According to a further aspect of the embodiments of the present disclosure, an instance segmentation model training apparatus is provided, comprising:

[0022] a second feature extraction module configured to perform feature extraction on a current training frame in a training video to obtain a current training feature map of the current training frame;

[0023] a third determination module configured to determine a reference training instance feature sequence based on respective historical training instance feature sequences of at least one historical training frame before the current training frame in the training video;

[0024] a fourth determination module configured to determine a current training instance feature sequence of the current training frame based on the current training feature map obtained by the second feature extraction module, the reference training instance feature sequence determined by the third determination module, and a historical training instance candidate box information sequence corresponding to a predetermined training frame in the at least one historical training frame, wherein the historical training instance candidate box information sequence comprises respective candidate box information of a plurality of historical instance candidate boxes in a historical training feature map of the predetermined training frame;

[0025] a second generation module configured to generate a video instance segmentation result of the current training frame via an initial instance segmentation model based on the current training feature map obtained by the second feature extraction module and the current training instance feature sequence determined by the fourth determination module;

[0026] a training module configured to train the initial instance segmentation model based on the video instance segmentation result of the current training frame generated by the second generation module, the annotation information of the current training frame, and the annotation information of the predetermined training frame;

[0027] a fifth determination module configured to determine the initial instance segmentation model after training as an instance segmentation model in response to the initial instance segmentation model after training satisfying a preset training end condition.

[0028] According to a further aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program for executing the above-mentioned video instance segmentation method or instance segmentation model training method.

[0029] According to a further aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0030] a processor;

[0031] a memory for storing instructions executable by the processor;

[0032] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the video instance segmentation method or the instance segmentation model training method.

[0033] The video instance segmentation method, the instance segmentation model training method, the device, the medium and the electronic device provided by the embodiments of the present disclosure can perform feature extraction on the current frame in the to-be-processed video to obtain a current feature map of the current frame, determine a reference instance feature sequence based on the historical instance feature sequences corresponding to at least one historical frame before the current frame in the to-be-processed video, and use the current feature map, the reference instance feature sequence and a historical instance candidate box information sequence corresponding to a predetermined frame in the at least one historical frame to determine a current instance feature sequence of the current frame. In this way, the determination process of the current instance feature sequence has effectively used the information propagated from the historical frame. By referring to the current feature map and the current instance feature sequence, the instance segmentation model can efficiently and reliably generate a video instance segmentation result of the current frame without performing an explicit association step, thereby reducing the complexity of the video instance segmentation algorithm and improving the video instance segmentation speed. BRIEF DESCRIPTION OF DRAWINGS

[0034] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. The drawings provided in the present disclosure are used to provide further understanding of the embodiments of the present disclosure and constitute a part of the specification, which serve to explain the present disclosure together with the embodiments of the present disclosure, but do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally indicate the same components or steps throughout the drawings.

[0035] Figure 1 FIG. 1 is a flowchart of a video instance segmentation method provided by an example embodiment of the present disclosure.

[0036] Figure 2 FIG. 2 is a system block diagram of a system for implementing video instance segmentation in an embodiment of the present disclosure.

[0037] Figure 3 FIG. 3 is a flowchart of a video instance segmentation method provided by another example embodiment of the present disclosure.

[0038] Figure 4 FIG. 4 is a flowchart of a video instance segmentation method provided by still another example embodiment of the present disclosure.

[0039] Figure 5 FIG. 5 is a workflow diagram of an instance query-object box propagation module in an embodiment of the present disclosure.

[0040] Figure 6 FIG. 6 is a flowchart of an instance segmentation model training method provided by an example embodiment of the present disclosure.

[0041] Figure 7 is a flowchart of an example instance segmentation model training method provided by another example embodiment of the present disclosure.

[0042] Figure 8 is a schematic diagram of a calculation principle of a frame loss value in an embodiment of the present disclosure.

[0043] Figure 9 is a schematic diagram of a label assignment strategy in a model training process in an embodiment of the present disclosure.

[0044] Figure 10 is a flowchart of a video instance segmentation device provided by an example embodiment of the present disclosure.

[0045] Figure 11 is a flowchart of a video instance segmentation device provided by another example embodiment of the present disclosure.

[0046] Figure 12 is a flowchart of an instance segmentation model training device provided by an example embodiment of the present disclosure.

[0047] Figure 13 is a flowchart of an instance segmentation model training device provided by another example embodiment of the present disclosure.

[0048] Figure 14 is a structural diagram of an electronic device provided by an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] Hereinafter, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein.

[0050] It should be noted that: unless otherwise specified, the relative arrangement, numerical expression and numerical value of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0051] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices, etc., and do not represent a specific technical meaning, nor necessarily represent a logical sequence.

[0052] It should also be understood that in the embodiments of the present disclosure, "a plurality of" can mean two or more, and "at least one" can mean one, two or more.

[0053] It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, it can be understood as one or more in general, without explicit limitation or in the context of the opposite indication given before and after.

[0054] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In the present disclosure, the character " / " generally represents that the front and rear associated objects are in an "or" relationship.

[0055] It should also be understood that the description of the various embodiments of the present disclosure focuses on the differences between the various embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0056] At the same time, it should be understood that, for the convenience of description, the size of each part shown in the drawings is not drawn according to the actual proportional relationship.

[0057] The following description of at least one example embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0058] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered part of the specification where appropriate.

[0059] It should be noted that similar reference numbers and letters refer to similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0060] Embodiments of the present disclosure can be applied to terminal devices, computer systems, servers, and other electronic devices, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, and the like.

[0061] Electronic devices such as terminal devices, computer systems, servers, and the like can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. Computer systems / servers can be practiced in distributed cloud-computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud-computing environment, program modules can be located in local or remote computer system storage media including storage devices.

[0062] Summary

[0063] Video instance segmentation is an important problem in the field of computer vision, and can be used to predict the semantic class of each pixel point in each frame of a video, distinguish different objects (also referred to as different instances) belonging to the same class, and associate the same object in different frames.

[0064] In the process of implementing the present disclosure, the inventors have found that there are generally two kinds of current video instance segmentation schemes, namely a frame-by-frame processing video instance segmentation scheme and a segment-by-segment processing video instance segmentation scheme. The principle of the frame-by-frame processing video instance segmentation scheme is to first obtain instance prediction results of each frame through a picture instance segmentation model, and then perform cross-frame instance association through an additional multi-target tracking model. The principle of the segment-by-segment processing video instance segmentation scheme is to first divide a video into multiple overlapping segments to obtain instance prediction results of each segment, and then associate these segments through an instance matching algorithm.

[0065] When the segment-by-segment processing video instance segmentation scheme is used, assuming that the total number of frames of pictures in a video is 9 frames, the first to third frames of the 9 frames can be taken as one segment, the third to fifth frames of the 9 frames can be taken as another segment, the fifth to seventh frames of the 9 frames can be taken as another segment, and the seventh to ninth frames of the 9 frames can be taken as another segment, thereby a total of four segments can be obtained. By obtaining instance prediction results corresponding to the four segments respectively, and then further associating the segments through an instance matching algorithm, instance segmentation of the video can be achieved.

[0066] It can be easily seen that, whether it is the frame-by-frame processing video instance segmentation scheme or the segment-by-segment processing video instance segmentation scheme, an explicit association step needs to be performed, which increases the algorithm complexity and reduces the video instance segmentation speed.

[0067] Exemplary method

[0068] Figure 1FIG. 1 is a flowchart of a method for video instance segmentation according to an example embodiment of the present disclosure. Figure 1 The method shown includes steps 110, 120, 130 and 140, which are described below.

[0069] In step 110, feature extraction is performed on a current frame in the video to be processed to obtain a current feature map of the current frame.

[0070] It should be noted that the instance segmentation model in the embodiments of the present disclosure can be obtained in advance through model training, such as Figure 2 As shown, the instance segmentation model can include a feature extraction subnetwork, an instance query and object box propagation subnetwork, and an instance decoding subnetwork; wherein the feature extraction subnetwork includes but is not limited to a convolutional neural network such as a Residual Network (ResNet).

[0071] Optionally, the video to be processed can be obtained based on video data collected by a camera (such as a front-view camera, a rear-view camera, etc.) arranged on the first vehicle, for example, the video data can be converted by a video frame collection module in the video frame collection device to obtain the video to be processed including consecutive video frames. Figure 2

[0072] Optionally, the current frame can be any frame in the video to be processed except the first frame. By using the feature extraction subnetwork to perform feature extraction on the current frame, a feature representation of the current frame can be obtained, which can be in the form of a feature map. At this time, the feature representation of the current frame can be used as the current feature map of the current frame.

[0073] In specific implementation, the remaining frames in the video to be processed except the first frame can be sequentially used as the current frame, for example, the second frame in the video to be processed can be used as the current frame first, and then the third frame in the video to be processed can be used as the current frame after the video instance segmentation result of the second frame is obtained by executing steps 110-140, and then the fourth frame in the video to be processed can be used as the current frame after the video instance segmentation result of the third frame is obtained by executing steps 110-140, and so on.

[0074] In step 120, a reference instance feature sequence is determined based on historical instance feature sequences corresponding to at least one historical frame before the current frame in the video to be processed.

[0075] ​It should be noted that when a first frame in the to-be-processed video is taken as a history frame, an instance feature sequence learned in a model training stage in the subsequent text can be taken as a history instance feature sequence corresponding to the first frame; when a non-first frame in the to-be-processed video is taken as a history frame, when the non-first frame is taken as a current frame, a current instance feature sequence obtained by performing steps 110 to 140 can be acquired, and the acquired current instance feature sequence is taken as a history instance feature sequence corresponding to the non-first frame.

[0076] It should be noted that each instance feature sequence in the embodiments of the present disclosure can include N instance features arranged in sequence; wherein N can be a preset maximum instance quantity in each frame, for example, 20, 30 or other numerical values; each instance feature can include feature values corresponding to a plurality of feature dimensions respectively, and the plurality of feature dimensions can be C feature dimensions specifically, and C can be 128, 256 or other numerical values.

[0077] If the total quantity of the history frames located before the current frame in the to-be-processed video is greater than or equal to T (which can be a preset stack length), in step 120, the history instance feature sequences corresponding to the continuous T history frames located before the current frame and next to the current frame in the to-be-processed video can be acquired, and the reference instance feature sequence can be determined based on the acquired history instance feature sequences.

[0078] If the total quantity of the history frames located before the current frame in the to-be-processed video is less than T, in step 120, the history instance feature sequences corresponding to all the history frames located before the current frame in the to-be-processed video can be acquired, and the reference instance feature sequence can be determined based on the acquired history instance feature sequences.

[0079] Optionally, the reference instance feature sequence can be determined via the instance query and object box propagation sub-network based on the acquired history instance feature sequences.

[0080] In step 130, the current instance feature sequence of the current frame can be determined based on the current feature map, the reference instance feature sequence and the history instance candidate box information sequence corresponding to a predetermined frame in the at least one history frame; wherein the history instance candidate box information sequence includes candidate box information corresponding to a plurality of history instance candidate boxes in the history feature map of the predetermined frame.

[0081] Optionally, the predetermined frame can be a last frame in the at least one history frame. In some cases, the predetermined frame can also be a second last frame or other frame in the at least one history frame, and for the convenience of understanding, the embodiments of the present disclosure are all illustrated taking the case that the predetermined frame is the last frame in the at least one history frame as an example.

[0082] When any non-first frame in the video to be processed is taken as the predetermined frame, before step 130, the historical feature map of the predetermined frame and the historical instance feature sequence corresponding to the predetermined frame can be provided as input data to the instance decoding subnetwork, and the instance decoding subnetwork can perform decoding processing based on the input data to obtain the video instance segmentation result of the predetermined frame; wherein the historical instance feature sequence corresponding to the predetermined frame can include N historical instance features, and the video instance segmentation result of the predetermined frame can include N historical instance prediction information corresponding to the N historical instance features.

[0083] Optionally, each historical instance prediction information can include at least one of the following: instance category information, instance location information, and instance tracking identification information; wherein each instance location information can include four sets of corner point coordinates, each set of corner point coordinates can represent a corner point, and a rectangular frame can be determined by the four sets of corner point coordinates; and each instance tracking identification information can be a tracking ID. In this way, it can be considered that each historical instance prediction information represents the candidate box information corresponding to a historical instance candidate box in the historical feature map of the predetermined frame, and thus the candidate box information corresponding to the N historical instance candidate boxes in the historical feature map of the predetermined frame can be obtained, thereby forming the historical instance candidate box information sequence corresponding to the predetermined frame.

[0084] When the first frame in the video to be processed is taken as the predetermined frame, an instance candidate box information sequence learned in the model training stage can be taken as the historical instance candidate box information sequence corresponding to the predetermined frame.

[0085] In step 130, based on the current feature map determined in step 110, the reference instance feature sequence determined in step 120, and the historical instance candidate box information sequence obtained in the above manner, the current instance feature sequence of the current frame can be determined via the instance query and object frame propagation module.

[0086] In step 140, based on the current feature map and the current instance feature sequence, the video instance segmentation result of the current frame can be generated via the instance segmentation model.

[0087] In step 140, the current feature map and the current instance feature sequence can be provided as input data to the instance decoding subnetwork, and the instance decoding subnetwork can perform decoding processing based on the input data to obtain the video instance segmentation result of the current frame. The composition of the video instance segmentation result of the current frame can refer to the related description of the video instance segmentation result of the predetermined frame in the foregoing, and will not be described here again.

[0088] Based on the video instance segmentation method provided in the above embodiments of this disclosure, feature extraction can be performed on the current frame of the video to be processed to obtain the current feature map of the current frame. Based on the historical instance feature sequences corresponding to at least one historical frame in the video to be processed that is located before the current frame, a reference instance feature sequence is determined. The current feature map, the reference instance feature sequence, and the historical instance candidate box information sequence corresponding to the predetermined frame in at least one historical frame are used to determine the current instance feature sequence of the current frame. In this way, the process of determining the current instance feature sequence has effectively utilized the information propagated from the historical frames. By referring to the current feature map and the current instance feature sequence, the video instance segmentation result of the current frame can be generated efficiently and reliably through the instance segmentation model without having to perform an explicit association step. This reduces the complexity of the video instance segmentation algorithm and improves the speed of video instance segmentation.

[0089] exist Figure 1 Based on the illustrated embodiments, as Figure 3 As shown, step 120 includes steps 1201, 1203 and 1205.

[0090] Step 1201: Determine the first similarity between at least one historical instance feature in the historical instance feature set and the first historical instance feature, and obtain at least one first similarity; wherein, the first historical instance feature is a historical instance feature in the historical instance feature sequence corresponding to a predetermined frame, the historical instance feature set includes: the first historical instance feature, and a historical instance feature in the historical instance feature sequence corresponding to each non-predetermined frame in at least one historical frame, and any two historical instance features in the historical instance feature set are in the same relative position in their respective historical instance feature sequences.

[0091] It should be noted that at least one historical frame corresponds one-to-one with at least one historical instance feature sequence. Since any two historical instance features in the historical instance feature set have the same relative position in their respective historical instance feature sequences, if the first historical instance feature is the i-th historical instance feature in the historical instance feature sequence corresponding to the predetermined frame, the historical instance feature set may include: the i-th historical instance feature in each of the at least one historical instance feature sequences.

[0092] In step 1201, for each historical instance feature in the historical instance feature set, the instance query and object box propagation subnetwork can calculate the first similarity between the historical instance feature and the first historical instance feature according to a preset feature similarity algorithm, thereby obtaining at least one first similarity corresponding to at least one historical instance feature in the historical instance feature set; wherein, the preset feature similarity algorithm includes, but is not limited to, cosine similarity algorithm, Euclidean distance algorithm, etc.

[0093] At step 1203, a first reference instance feature is determined based on the historical instance feature set and the at least one first similarity.

[0094] In an optional implementation, step 1203 comprises:

[0095] Based on the at least one first similarity, a first weight is respectively assigned to at least one historical instance feature in the historical instance feature set, to obtain at least one first weight.

[0096] The at least one historical instance feature in the historical instance feature set is weighted by using the at least one first weight, to obtain a first weighted instance feature.

[0097] The first weighted instance feature and the first historical instance feature are fused by feature, to obtain the first reference instance feature.

[0098] Optionally, a one-to-one correspondence relationship can exist between the at least one first weight and the at least one first similarity, and a one-to-one correspondence relationship can also exist between the at least one first weight and the at least one historical instance feature in the historical instance feature set.

[0099] In an example, the at least one historical instance feature in the historical instance feature set is six historical instance features, which are c1, c2, c3, c4, c5 and c6 in sequence, the first similarity corresponding to c1 to c6 is g1, g2, g3, g4, g5 and g6 in sequence, and the first weight corresponding to c1 to c6 is w1, w2, w3, w4, w5 and w6 in sequence. Then, w1 to w6 can be calculated by using the following formula:

[0100] w1 = e g1 / (e g1 +e g2 +e g3 +e g4 +e g5 +e g6 )

[0101] w2 = e g2 / (e g1 +e g2 +e g3 +e g4 +e g5 +e g6 )

[0102] w3 = e g3 / (e g1 +e g2 +e g3 +e g4 +e g5 +eg6 )

[0103] w4 = e g4 / (e g1 +e g2 +e g3 +e g4 +e g5 +e g6 )

[0104] w5 = e g5 / (e g1 +e g2 +e g3 +e g4 +e g5 +e g6 )

[0105] w6 = e g6 / (e g1 +e g2 +e g3 +e g4 +e g5 +e g6 )

[0106] Alternatively, w1 to w6 can be calculated using the following formulae:

[0107] w1 = g1 / (g1+g2+g3+g4+g5+g6)

[0108] w2 = g2 / (g1+g2+g3+g4+g5+g6)

[0109] w3 = g3 / (g1+g2+g3+g4+g5+g6)

[0110] w4 = g4 / (g1+g2+g3+g4+g5+g6)

[0111] w5 = g5 / (g1+g2+g3+g4+g5+g6)

[0112] w6 = g6 / (g1+g2+g3+g4+g5+g6)

[0113] Next, c1 to c6 can be weighted and summed using w1 to w6 to obtain a first weighted instance feature. Assuming that the first weighted instance feature is denoted as c, the first weighted instance feature can be calculated using the following formula:

[0114] c = w1*c1+w2*c2+w3*c3+w4*c4+w5*c5+w6*c6

[0115] It should be noted that when determining w1 to w6 above, w1 to w6 can also be directly set to satisfy:

[0116] w1 = g1

[0117] w2 = g2

[0118] w3 = g3

[0119] w4 = g4

[0120] w5 = g5

[0121] w6 = g6

[0122] In this case, c1 to c6 can be weighted and averaged by using w1 to w6 to obtain c as the first weighted instance feature. At this time, the first weighted instance feature can be calculated by using the following formula:

[0123]

[0124] After that, the first reference instance feature can be obtained by fusing the first weighted instance feature and the first historical instance feature. Assuming that the first reference instance feature is represented as c6 * , the first reference instance feature can be calculated by using the following formula:

[0125] c6 * = c + c6

[0126] In this embodiment, the reference to at least one first similarity can reasonably distribute the weights, and by using at least one first weight obtained by the weight distribution, at least one historical instance feature in the historical instance feature set can be weighted, and then the first reference instance feature can be efficiently and reliably obtained by fusing the weighted result and the first historical instance feature. In addition, the determination process of the first reference instance feature effectively uses the information propagated from the historical frame, so as to effectively ensure the feature quality of the first reference instance feature.

[0127] Step 1205, determining a reference instance feature sequence based on the first reference instance feature, and the relative position of the first reference instance feature in the reference instance feature sequence is consistent with the relative position of the first historical instance feature in the historical instance feature sequence corresponding to the predetermined frame.

[0128] It should be noted that the above describes the determination manner of the first reference instance feature corresponding to the first historical instance feature, and the remaining historical instance features in the historical instance feature sequence corresponding to the predetermined frame can also be determined in a similar manner. By sequentially arranging the obtained reference instance features, a reference instance feature sequence can be formed.

[0129] The first reference instance feature is the i-th reference instance feature in the reference instance feature sequence, if the first historical instance feature is the i-th historical instance feature in the historical instance feature sequence corresponding to the predetermined frame.

[0130] Optionally, the reference instance feature sequence can be determined according to the following formula:

[0131]

[0132] wherein i∈[1, N], denotes the i-th reference instance feature in the reference instance feature sequence, may be the first reference instance feature, denotes one historical instance feature in the historical instance feature set, denotes the first weight corresponding to denotes the i-th historical instance feature in the historical instance feature sequence corresponding to the predetermined frame, may be the first historical instance feature.

[0133] In the embodiments of the present disclosure, by respectively determining the first similarity between at least one historical instance feature in the historical instance feature set and the first historical instance feature, the at least one first similarity obtained and the historical instance feature set can be referred to, the first reference instance feature can be efficiently and reasonably determined, the reference instance feature sequence with excellent feature quality can be obtained, and the obtained reference instance feature sequence can be used for determining the current instance feature sequence. In this way, by effectively using the information propagated from the historical frame, the reliability of the current instance feature sequence can be better guaranteed, thereby being beneficial to guaranteeing the accuracy and reliability of the video instance segmentation result of the current frame.

[0134] Based on the embodiment shown in Figure 1 , as shown in Figure 4 , the step 130 comprises a step 1301, a step 1303, a step 1305 and a step 1307.

[0135] The step 1301 comprises determining a plurality of instance candidate boxes in the current feature map according to the historical instance candidate box information sequence.

[0136] ​It can be known from the introduction of the historical instance candidate box information sequence above that the historical instance candidate box information sequence can include N instance position information, each of which can include four groups of corner point coordinates, so that the N instance position information sequentially arranged in the historical instance candidate box information sequence can sequentially determine N instance candidate boxes in the current feature map; and the N instance candidate boxes and the N reference instance features in the reference instance feature sequence can be in a one-to-one correspondence.

[0137] At step 1303, pixel features of the plurality of pixel points in the first instance candidate box corresponding to the second reference instance feature in the reference instance feature sequence are determined, to obtain a plurality of pixel features.

[0138] It should be noted that the second reference instance feature can be any reference instance feature in the reference instance feature sequence.

[0139] In step 1303, the first instance candidate box corresponding to the second reference instance feature can be determined from the plurality of instance candidate boxes, and then the pixel features of the plurality of pixel points in the first instance candidate box are determined, thereby obtaining the plurality of pixel features corresponding to the plurality of pixel points one by one; similarly to each instance feature, each pixel feature can also include a plurality of feature values corresponding to a plurality of feature dimensions.

[0140] At step 1305, the first current instance feature is determined based on the second reference instance feature and the plurality of pixel features.

[0141] In an optional implementation, step 1305 includes:

[0142] A second similarity between the plurality of pixel features and the second reference instance feature is respectively determined, to obtain a plurality of second similarities;

[0143] Based on the plurality of second similarities, a second weight is respectively assigned to the plurality of pixel points, to obtain a plurality of second weights;

[0144] The plurality of pixel features are weighted using the plurality of second weights, to obtain the first current instance feature.

[0145] Optionally, for each pixel feature in the plurality of pixel features, a second similarity between the pixel feature and the second reference instance feature can be calculated according to a preset feature similarity algorithm, thereby obtaining the plurality of second similarities corresponding to the plurality of pixel features one by one.

[0146] Next, weights can be assigned based on multiple second similarities to obtain multiple second weights, and multiple second weights can be used to weight multiple pixel features to obtain the first current instance feature. The specific weight assignment method and weighting method can be referred to the relevant description of the implementation method of step 1203 above, and will not be repeated here.

[0147] In this way, by determining the second similarity and reasonably allocating weights based on the determined second similarities, and then combining weighted processing, the first current instance feature can be obtained efficiently and reliably. Furthermore, the first current instance feature can be considered as the result obtained after modifying the second reference instance feature, and the feature quality of the first current instance feature can be well guaranteed.

[0148] Step 1307: Based on the first current instance feature, determine the current instance feature sequence of the current frame, wherein the relative position of the first current instance feature in the current instance feature sequence is consistent with the relative position of the second reference instance feature in the reference instance feature sequence.

[0149] It should be noted that the specific implementation of step 1307 can be referred to the relevant introduction of step 1205 above, and will not be repeated here.

[0150] Due to the continuity of video, objects in the previous frame will reappear in similar positions in the next frame. In view of this, in the embodiments of this disclosure, multiple instance candidate boxes can be determined in the current feature map according to the historical instance candidate box information sequence, and the pixel features of the pixels in the determined instance candidate boxes can be used to correct the reference instance features in the reference instance feature sequence, thereby obtaining a current instance feature sequence with excellent feature quality. The obtained current instance feature sequence can be used to generate the video instance segmentation result of the current frame, thereby helping to ensure the accuracy and reliability of the video instance segmentation result of the current frame.

[0151] In an optional example, for the video to be processed, the feature representation of the current frame (equivalent to the current feature map mentioned above) can be extracted by a convolutional neural network. The current feature map is then input into the instance query and object box propagation subnetwork. Using the information propagated from historical frames, the instance query and object box propagation subnetwork can generate the current instance feature sequence.

[0152] Specifically, such as Figure 5 As shown, the instance query and object bounding box propagation subnetwork can utilize the object bounding box detection results of the previous frame. t-1 (equivalent to the historical instance candidate box information sequence mentioned above), in the current feature map x t Extracting region features (ROI) of candidate regions that may contain objects. tcorresponding to the reference instance feature sequence, and the instance query propagation subnetwork can determine the reference instance feature sequence by using the historical instance feature sequence corresponding to each of at least one historical frame before the current frame and the region feature roi t , and correct the reference instance feature sequence to obtain the current instance feature sequence q t For any reference instance feature in the reference instance feature sequence, a plurality of second weights can be obtained based on the plurality of pixel features and the reference instance feature by using a preset feature similarity algorithm, and the plurality of pixel features can be weighted by using the plurality of second weights, and the weighted result can be considered as the correction result of the reference instance feature, and the correction result can be considered as the current instance feature corresponding to the reference instance feature in the current instance feature sequence q t .

[0153] Then, the current feature map and the current instance feature sequence q t can be input into the instance decoding subnetwork for decoding, so as to generate the video instance segmentation result of the current frame. It should be noted that in the embodiments of the present disclosure, the plurality of instance features in any instance feature sequence are sequentially arranged, and the relative positions of the instance features of the same instance in the current instance feature sequences corresponding to different frames are consistent, so that the same instance is decoded at the same bit sequence on different frames, for example, the t in the current instance feature sequence q corresponds to decodes the same instance, so that implicit instance association can be realized.

[0154] Any video instance segmentation method provided by the embodiments of the present disclosure can be executed by any appropriate device with data processing capability, including but not limited to: terminal devices and servers, etc. Alternatively, any video instance segmentation method provided by the embodiments of the present disclosure can be executed by a processor, such as a processor executing any video instance segmentation method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. The following will not be described again.

[0155] Figure 6 is a flowchart of an instance segmentation model training method provided by an exemplary embodiment of the present disclosure. Figure 6 The method shown includes steps 610, 620, 630, 640, 650 and 660, which will be described below.

[0156] Step 610: performing feature extraction on a current training frame in a training video to obtain a current training feature map of the current training frame.

[0157] At step 620, a reference training instance feature sequence is determined based on respective historical training instance feature sequences of at least one historical training frame located before the current training frame in the training video.

[0158] At step 630, a current training instance feature sequence of the current training frame is determined based on the current training feature map, the reference training instance feature sequence, and a historical training instance candidate box information sequence corresponding to a predetermined training frame in the at least one historical training frame, wherein the historical training instance candidate box information sequence comprises candidate box information corresponding to each of a plurality of historical instance candidate boxes in a historical training feature map of the predetermined training frame.

[0159] At step 640, a video instance segmentation result of the current training frame is generated via the initial instance segmentation model based on the current training feature map and the current training instance feature sequence.

[0160] It should be noted that the specific implementation process of steps 610 to 640 can be referred to the related introduction of steps 110 to 140, which will not be repeated here.

[0161] At step 650, the initial instance segmentation model is trained based on the video instance segmentation result of the current training frame, the annotation information of the current training frame, and the annotation information of the predetermined training frame.

[0162] It should be noted that the initial instance segmentation model refers to the instance segmentation model to be trained, so the initial instance segmentation model can also include a feature extraction subnetwork, an instance query and object box propagation subnetwork, and an instance decoding subnetwork. The composition of the video instance segmentation result of the current training frame can be referred to the related description of the video instance segmentation result of the predetermined frame, which will not be repeated here.

[0163] Optionally, the annotation information of the current training frame can include instance category information, instance position information, and instance tracking identification information of each of a plurality of instances in the current training frame.

[0164] Similarly, the annotation information of the predetermined training frame can include instance category information, instance position information, and instance tracking identification information of each of a plurality of instances in the predetermined training frame.

[0165] In step 650, multiple iterations can be performed with reference to the video instance segmentation result of the current training frame, the annotation information of the current training frame, and the annotation information of the predetermined training frame. In each iteration, the initial instance segmentation model is updated to optimize the model parameters of the initial instance segmentation model.

[0166] At step 660, in response to the trained initial instance segmentation model meeting a preset training end condition, the trained initial instance segmentation model is determined as the instance segmentation model.

[0167] If the trained initial instance segmentation model is detected to converge at a certain moment, it can be determined that the trained initial instance segmentation model meets the preset training end condition, and at this time, the trained initial instance segmentation model can be directly determined as the trained instance segmentation model.

[0168] Of course, the preset training end condition is not limited to this, for example, the trained initial instance segmentation model can also be determined to meet the preset training end condition when the number of iteration processing times reaches a preset number (for example, 10000 times, 20000 times).

[0169] In the embodiments of the present disclosure, feature extraction can be performed on the current training frame in the training video to obtain a current training feature map of the current training frame, a reference training instance feature sequence is determined based on the respective historical training instance feature sequences of at least one historical training frame located before the current training frame in the training video, and the current training feature map, the reference training instance feature sequence, and the historical training instance candidate box information sequence corresponding to the predetermined training frame in the at least one historical training frame are used for determination of a current training instance feature sequence of the current training frame. In this way, the determination process of the current training instance feature sequence effectively uses the information propagated from the historical training frame. By referring to the current training feature map and the current training instance feature sequence, the initial instance segmentation model can efficiently and reliably generate a video instance segmentation result of the current training frame. Then, by using the video instance segmentation result of the current training frame, the annotation information of the current training frame, and the annotation information of the predetermined training frame for training of the initial instance segmentation model, a trained instance segmentation model can be efficiently and reliably obtained. In the actual use stage of the model, the video instance segmentation result can be efficiently and reliably generated without performing an explicit association step, thereby reducing the complexity of the video instance segmentation algorithm and improving the video instance segmentation speed.

[0170] Based on the embodiment shown in Figure 6 As shown in Figure 7 Step 650 includes step 6501, step 6503, step 6505, step 6507, and step 6509.

[0171] Step 6501, based on the video instance segmentation result of the current training frame, determines a current instance prediction information sequence of the current training frame; wherein the current instance prediction information sequence includes a plurality of current instance prediction information, and the plurality of current instance prediction information corresponds to a plurality of instance candidate boxes.

[0172] Optionally, the video instance segmentation result of the current training frame can include a current instance prediction information sequence of the current training frame, and the current instance prediction information sequence can include N instance prediction information arranged in sequence. The composition of the instance prediction information can refer to the related description of the historical instance prediction information above, and will not be described here.

[0173] In step 6501, the current instance prediction information sequence can be directly extracted from the video instance segmentation result of the current training frame.

[0174] In step 6503, based on the annotation information of the current training frame, the annotation information of the predetermined training frame, and the historical tracking identifier allocation situation information of the historical instance prediction information sequence of the predetermined training frame, the tracking identifier allocation of the current instance prediction information sequence is performed.

[0175] Optionally, the historical tracking identifier allocation situation information of the historical instance prediction information sequence of the predetermined training frame can be used to indicate which historical instance prediction information in the N instance prediction information included in the historical instance prediction information sequence is allocated with a historical tracking identifier, and the specific historical tracking identifier is allocated.

[0176] In an optional implementation, step 6503 includes:

[0177] In response to determining, based on the historical tracking identifier allocation situation information, that a first historical instance prediction information corresponding to a first current instance prediction information in the current instance prediction information sequence is not allocated with a tracking identifier, the first current instance prediction information is added to the new instance prediction information set; wherein the first current instance prediction information and the first historical instance prediction information correspond to the fact that the relative position of the first historical instance prediction information in the historical instance prediction information sequence is consistent with the relative position of the first current instance prediction information in the current instance prediction information sequence.

[0178] In response to determining, based on the historical tracking identifier allocation situation information, that the first historical instance prediction information is allocated with a first tracking identifier different from the preset tracking identifier, and the first tracking identifier exists in the annotation information of the current training frame, the first tracking identifier is allocated to the first current instance prediction information.

[0179] In response to determining, based on the historical tracking identifier allocation situation information, that the first historical instance prediction information is allocated with the first tracking identifier, and the first tracking identifier does not exist in the annotation information of the current training frame, the preset tracking identifier is allocated to the first current instance prediction information.

[0180] The new instance annotation information set is determined by comparing the annotation information of the current training frame with the annotation information of the predetermined training frame.

[0181] The newly-added instance annotation information set is matched with the newly-added instance prediction information set, and based on a matching result, a tracking identifier is assigned to at least part of the current instance prediction information in the newly-added instance prediction information set.

[0182] Optionally, the newly-added instance prediction information set can be represented as The newly-added instance annotation information set can be represented as The preset tracking identifier can be a preset background identifier.

[0183] It should be noted that the first current instance prediction information can be any current instance prediction information in the current instance prediction information sequence. Since the first current instance prediction information corresponds to the first historical instance prediction information, if the first current instance prediction information is the i th current instance prediction information in the current instance prediction information sequence, the first historical instance prediction information can be the i th historical instance prediction information in the historical instance prediction information sequence.

[0184] In this implementation, by analyzing the historical tracking identifier assignment information, it can be determined whether the first historical instance prediction information is assigned a tracking identifier, and the specific representation of the assigned tracking identifier.

[0185] If the first historical instance prediction information is not assigned a tracking identifier, the first current instance prediction information can be added to .

[0186] If the first historical instance prediction information is assigned a first tracking identifier different from the preset tracking identifier, and the first tracking identifier exists in the annotation information of the current training frame, it indicates that the instance corresponding to the first historical instance prediction has appeared in the predetermined training frame, and the instance still exists in the current sequence frame, and then the first tracking identifier can be assigned to the first current instance prediction information, so that the first current instance prediction information reuses the tracking identifier that has been assigned to it before.

[0187] If the first historical instance prediction information is assigned the first tracking identifier, and the first tracking identifier does not exist in the annotation information of the current training frame, it indicates that the instance corresponding to the first historical instance prediction has appeared in the predetermined training frame, and the instance does not exist in the current sequence frame, that is, the instance disappears in the current training frame, and then the preset tracking identifier can be assigned to the first current instance prediction information, to represent that the first current instance prediction information corresponds to the background part of the image of the current training frame.

[0188] In addition, the annotation information of the current training frame and the annotation information of the predetermined training frame can also be compared to determine which instance annotation information is added to the annotation information of the current training frame compared with the annotation information of the predetermined training frame, and the set of the instance annotation information can be used as the newly-added instance annotation information set.

[0189] Afterwards, the newly-added instance prediction information set and the newly-added instance annotation information set can be matched through a Hungarian matching algorithm to determine which instance prediction information in the newly-added instance prediction information set can be matched with the instance prediction information in the newly-added instance annotation information set.

[0190] Suppose that the newly-added instance prediction information set includes newly-added instance prediction information a1, newly-added instance prediction information a2, and newly-added instance prediction information a3, the newly-added instance annotation information set includes newly-added instance annotation information b1 and newly-added instance annotation information b2, and b1 can be matched with a1 and b2 can be matched with a3, then the tracking identifier h1 annotated for b1 can be assigned to a1, the tracking identifier h2 annotated for b2 can be assigned to a3, and no tracking identifier is assigned to a2.

[0191] In this implementation, by reusing the tracking identifier previously assigned to the repeatedly appearing instance in the model training stage, assigning the tracking identifier representing the background to the instance that has appeared before but disappeared, and assigning appropriate tracking identifiers to the newly appearing instance based on the Hungarian matching algorithm, the instance segmentation model can accurately automatically assign appropriate tracking identifiers to each instance in each frame to associate the same instance in different frames in the actual use stage of the model, thereby better ensuring the accuracy and reliability of the video instance segmentation result.

[0192] In step 6505, based on the current tracking identifier assignment information of the current instance prediction information sequence, the multiple instance candidate boxes corresponding to the multiple current instance prediction information are divided into a positive sample box set and a negative sample box set.

[0193] In an optional implementation, step 6505 includes:

[0194] determining each current instance prediction information in the current instance prediction information sequence to which a tracking identifier different from the preset tracking identifier is assigned;

[0195] regarding a set composed of the instance candidate boxes corresponding to the determined each current instance prediction information as the positive sample box set, and regarding a set composed of the remaining instance candidate boxes as the negative sample box set.

[0196] Suppose that the current instance prediction information sequence includes 20 current instance prediction information, and only 5 current video instance prediction information in the 20 current video instance prediction information are respectively assigned a tracking identifier different from the preset tracking identifier, then a set composed of 5 instance candidate boxes corresponding to the 5 current video instance prediction information can be regarded as the positive sample box set, and a set composed of 15 instance candidate boxes corresponding to the remaining 15 current video instance prediction information can be regarded as the negative sample box set.

[0197] With this implementation, multiple instance candidate boxes can be efficiently and reliably divided into a positive sample box set and a negative sample box set, and each instance candidate box in the positive sample box set can be considered as a positive sample matched with the real annotation information, and each instance candidate box in the negative sample box set can be considered as a negative sample not matched with the real annotation information.

[0198] In some cases, the multiple instance candidate boxes can also have respective corresponding confidence values (which can be predicted by the initial instance prediction model), and in step 6505, the multiple instance candidate boxes can be divided into a positive sample box set and a negative sample box set in combination with the current tracking identification assignment information of the current instance prediction information sequence and the respective corresponding confidence values of the multiple instance candidate boxes.

[0199] In step 6507, a model loss value is determined based on the positive sample box set and the negative sample box set.

[0200] In an optional implementation, step 6507 includes:

[0201] From the negative sample box set, a third instance candidate box with the maximum intersection over union with the second instance candidate box in the positive sample box set is determined;

[0202] A center distance between the second instance candidate box and the third instance candidate box is determined.

[0203] A diagonal line length of the second instance candidate box is determined.

[0204] Based on the center distance, the diagonal line length, a first preset numerical value and a second preset numerical value, the model loss value is determined.

[0205] It should be noted that the second instance candidate box can be any instance candidate box in the positive sample box set.

[0206] Suppose the positive sample box set includes V1 instance candidate boxes, and the negative sample box set includes V2 instance candidate boxes, and V1 and V2 are both greater than 1, then the intersection over union between each instance candidate box in the negative sample box set and the second instance candidate box in the positive sample box set can be determined respectively, and thus V2 intersection over unions corresponding to the V2 instance candidate boxes in the negative sample box set can be obtained.

[0207] Next, the maximum numerical value of the V2 intersection over unions can be selected from the V2 intersection over unions, and the instance candidate box in the negative sample box set corresponding to the maximum numerical value can be taken as the third instance candidate box.

[0208] Then, the center distance between the second instance candidate box and the third instance candidate box, as well as the diagonal length of the second instance candidate box, can be calculated, and the center distance and diagonal length can be used to determine the model loss value.

[0209] Optionally, the model loss value is determined based on the center distance, diagonal length, a first preset value, and a second preset value, including:

[0210] The first value is obtained by exponentiation of the center distance;

[0211] The second value is obtained by exponentiation of the diagonal length.

[0212] Calculate the ratio of the first operand value to the second operand value;

[0213] Calculate the difference between the first preset value and the ratio;

[0214] Select the larger value between the difference and the second preset value;

[0215] The larger value is used to determine the model loss value.

[0216] Assuming the second instance candidate box is as follows Figure 8 solid line box b i (It can be considered as a ground truth box) As shown, the third instance candidate box is as follows: Figure 8 dashed box in As shown, the center distance can be expressed as The length of the diagonal can be expressed as d(b) i ).

[0217] Assuming exponentiation is a quadratic exponentiation, the first operand can be represented as: The second operand can be represented as d. 2 (b i The ratio of the first operand to the second operand can be expressed as:

[0218] Assuming the first preset value is denoted as β and the second preset value is 0, the interpolation between the first preset value and the ratio can be expressed as:

[0219] Assume the larger value is represented by L dedup The larger value can be calculated using the following formula:

[0220]

[0221] After obtaining the larger value, the larger value can be taken as the box loss value corresponding to the second instance candidate box, and in a similar manner, the box loss values corresponding to the respective remaining instance candidate boxes in the positive sample box set can also be obtained, so that a total of V1 box loss values can be obtained, and then the average of the plurality of box loss values can be calculated, and the calculated average can be taken as the model loss value.

[0222] In this way, the model loss value can be efficiently and reliably calculated through simple operation logics such as power operation, division operation, subtraction operation, and size comparison operation.

[0223] Of course, the determination manner of the model loss value is not limited thereto, for example, the model loss value can also be calculated through other types of loss functions, as long as the model loss value is negatively correlated with the center distance.

[0224] In this implementation, by determining the third instance candidate box with the maximum IoU with the second instance candidate box in the positive sample box set from the negative sample box set, and combining the center distance between the second instance candidate box and the third instance candidate box, the diagonal length of the second instance candidate box, and the first preset value and the second preset value, the model loss value can be efficiently and reliably determined, and when the determined model loss value is used for model training, it is beneficial to pull apart the center distance between the negative sample box and the real box, so that the center distance is greater than the first preset value, thereby facilitating the weakening of the redundant box aggregation situation.

[0225] In step 6509, based on the model loss value, the initial instance segmentation model is trained.

[0226] In step 6509, referring to the model loss value, the gradient descent algorithm (such as the stochastic gradient descent algorithm) can be used to adjust the parameters of the initial instance segmentation model.

[0227] In the embodiments of the present disclosure, referring to the video instance segmentation result of the current training frame, the current instance prediction information sequence of the current training frame can be efficiently and reliably determined, referring to the annotation information of the current training frame, the annotation information of the predetermined training frame, and the historical tracking identifier allocation information of the historical instance prediction information sequence of the predetermined training frame, the tracking identifier allocation for the current instance prediction information sequence can be reasonably performed, based on the obtained current tracking identifier allocation information of the current instance prediction information sequence, the plurality of instance candidate boxes corresponding to the plurality of current instance prediction information can be reasonably divided into the positive sample box set and the negative sample box set, so as to obtain the model loss value through the calculation of the box loss value for the training of the initial instance segmentation model, which is beneficial to reduce the possibility of the trained instance segmentation model to produce repeated prediction boxes, reduce redundancy, and improve the accuracy and reliability of the instance segmentation model.

[0228] In an optional example, after obtaining the current training feature map and the current training instance feature sequence of the current training frame, the current training feature map and the current training instance feature sequence can be provided as input data to an instance decoding subnetwork of the initial instance segmentation model, and the instance decoding subnetwork can perform decoding processing based on the input data to obtain a video instance segmentation result of the current training frame. The video instance segmentation result of the current training frame can include a current instance prediction information sequence, and the current instance prediction information sequence can include N current instance prediction information. The i-th current instance prediction information in the current instance prediction information sequence can be represented as Figure 9 i∈[1,N].

[0229] Suppose the current training feature map is the t-th frame training feature map, as shown in Figure 9 , it can be determined whether the instance corresponding to the current instance prediction information appears in the (t-1)-th frame training feature map (i.e., a historical training feature map of the predetermined training frame).

[0230] If the instance corresponding to the current instance prediction information appears in the (t-1)-th frame training feature map, the tracking ID corresponding to the instance in the (t-1)-th frame training feature map is ID1, and ID1 exists in the annotation information of the current training frame, the label assignment result of the (t-1)-th frame training feature map can be reused, that is, ID1 is assigned to the current instance prediction information

[0231] If the instance corresponding to the current instance prediction information appears in the (t-1)-th frame training feature map, the tracking ID corresponding to the instance in the (t-1)-th frame training feature map is ID1, and ID1 does not exist in the annotation information of the current training frame, ID2 as a preset tracking ID can be assigned to the current instance prediction information

[0232] If the instance corresponding to the current instance prediction information does not appear in the (t-1)-th frame training feature map, the current instance prediction information can be added to a new instance prediction information set , and subsequently, at least part of the new instance prediction information in the new instance prediction information set can be assigned a tracking ID by using a Hungarian matching algorithm.

[0233] After the assignment of the tracking ID is completed, the positive sample box set and the negative sample box set can be determined based on the assignment result, so that the positive sample box set and the negative sample box set are used for determining the box loss value, so as to optimize the parameters of the initial instance segmentation model by using the box loss value. ​

[0234] In summary, the embodiments of this disclosure can combine the processes of instance segmentation and instance association into one, with the two promoting each other, thereby improving the accuracy of the video instance segmentation algorithm and reducing its complexity.

[0235] Any instance segmentation model training method provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any instance segmentation model training method provided in this disclosure can be executed by a processor, such as by a processor executing any instance segmentation model training method mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0236] Exemplary apparatus

[0237] Figure 10 This is a schematic diagram of the structure of a video instance segmentation apparatus provided in an exemplary embodiment of the present disclosure. Figure 10 The apparatus shown includes a first feature extraction module 1010, a first determination module 1020, a second determination module 1030, and a first generation module 1040.

[0238] The first feature extraction module 1010 is used to extract features from the current frame of the video to be processed, and obtain the current feature map of the current frame.

[0239] The first determining module 1020 is used to determine a reference instance feature sequence based on the historical instance feature sequence corresponding to at least one historical frame in the video to be processed that is located before the current frame.

[0240] The second determining module 1030 is used to determine the current instance feature sequence of the current frame based on the current feature map obtained by the first feature extraction module 1010, the reference instance feature sequence determined by the first determining module 1020, and the historical instance candidate box information sequence corresponding to a predetermined frame in at least one historical frame; wherein, the historical instance candidate box information sequence includes: candidate box information corresponding to each of the multiple historical instance candidate boxes in the historical feature map of the predetermined frame.

[0241] The first generation module 1040 is used to generate a video instance segmentation result for the current frame based on the current feature map obtained by the first feature extraction module 1010 and the current instance feature sequence determined by the second determination module 1030, via an instance segmentation model.

[0242] In an optional example, such as Figure 11 As shown, the first determining module 1020 includes:

[0243] The first determining sub-module 10201 is configured to determine a first similarity between at least one historical instance feature in the historical instance feature set and the first historical instance feature respectively, to obtain at least one first similarity; wherein the first historical instance feature is one historical instance feature in the historical instance feature sequence corresponding to the predetermined frame, and the historical instance feature set includes: the first historical instance feature, and one historical instance feature in the historical instance feature sequence corresponding to each non-predetermined frame in the at least one historical frame, and any two historical instance features in the historical instance feature set have a same relative position in the historical instance feature sequence in which each of the two historical instance features is located.

[0244] The second determining sub-module 10203 is configured to determine the first reference instance feature based on the historical instance feature set and the at least one first similarity obtained by the first determining sub-module 10201.

[0245] The third determining sub-module 10205 is configured to determine the reference instance feature sequence based on the first reference instance feature determined by the second determining sub-module 10203, and the relative position of the first reference instance feature in the reference instance feature sequence is consistent with the relative position of the first historical instance feature in the historical instance feature sequence corresponding to the predetermined frame.

[0246] In an optional example, the second determining sub-module 10203 includes:

[0247] The first assigning unit is configured to assign a first weight to at least one historical instance feature in the historical instance feature set based on the at least one first similarity obtained by the first determining sub-module 10201, to obtain at least one first weight.

[0248] The first weighting unit is configured to weight at least one historical instance feature in the historical instance feature set by using the at least one first weight obtained by the first assigning unit, to obtain a first weighted instance feature.

[0249] The fusion unit is configured to perform feature fusion on the first weighted instance feature obtained by the first weighting unit and the first historical instance feature, to obtain the first reference instance feature.

[0250] In an optional example, as shown in Figure 11 The second determining module 1030 includes:

[0251] The fourth determining sub-module 10301 is configured to determine a plurality of instance candidate boxes in the current feature map according to the historical instance candidate box information sequence.

[0252] The fifth determining sub-module 10303 is configured to determine pixel features of a plurality of pixel points in the first instance candidate box corresponding to the second reference instance feature in the reference instance feature sequence determined by the third determining sub-module 10205, to obtain a plurality of pixel features.

[0253] The sixth determining sub-module 10305 is configured to determine the first current instance feature based on the second reference instance feature and the plurality of pixel features obtained by the fifth determining sub-module 10303.

[0254] The seventh determining sub-module 10307 is configured to determine a current instance feature sequence of the current frame based on the first current instance feature determined by the sixth determining sub-module 10305, and a relative position of the first current instance feature in the current instance feature sequence is consistent with a relative position of the second reference instance feature in the reference instance feature sequence.

[0255] In an optional example, the sixth determining sub-module 10305 includes:

[0256] The first determining unit is configured to determine a second similarity between each of the plurality of pixel features obtained by the fifth determining sub-module 10303 and the second reference instance feature, to obtain a plurality of second similarities.

[0257] The second assigning unit is configured to assign a second weight to each of the plurality of pixel points based on the plurality of second similarities obtained by the first determining unit, to obtain a plurality of second weights.

[0258] The second weighting unit is configured to weight the plurality of pixel features obtained by the fifth determining sub-module 10303 by using the plurality of second weights obtained by the second assigning unit, to obtain the first current instance feature.

[0259] Figure 12 FIG. 1 is a structural schematic diagram of a video instance segmentation device provided by an example embodiment of the present disclosure. Figure 12 The device shown includes a second feature extraction module 1210, a third determining module 1220, a fourth determining module 1230, a second generation module 1240, a training module 1250, and a fifth determining module 1260.

[0260] The second feature extraction module 1210 is configured to perform feature extraction on a current training frame in a training video, to obtain a current training feature map of the current training frame.

[0261] The third determining module 1220 is configured to determine a reference training instance feature sequence based on a plurality of historical training instance feature sequences respectively corresponding to at least one historical training frame before the current training frame in the training video.

[0262] The fourth determination module 1230 is configured to determine a current training instance feature sequence of the current training frame based on the current training feature map obtained by the second feature extraction module 1210, the reference training instance feature sequence determined by the third determination module 1220, and historical training instance candidate box information sequences corresponding to the predetermined training frames in the at least one historical training frame. The historical training instance candidate box information sequence includes candidate box information corresponding to each of the multiple historical instance candidate boxes in the historical training feature map of the predetermined training frame.

[0263] The second generation module 1240 is configured to generate a video instance segmentation result of the current training frame via the initial instance segmentation model based on the current training feature map obtained by the second feature extraction module 1210 and the current training instance feature sequence determined by the fourth determination module 1230.

[0264] The training module 1250 is configured to train the initial instance segmentation model based on the video instance segmentation result of the current training frame generated by the second generation module 1240, the annotation information of the current training frame, and the annotation information of the predetermined training frame.

[0265] The fifth determination module 1260 is configured to determine the initial instance segmentation model trained by the training module 1250 as the instance segmentation model in response to the initial instance segmentation model trained by the training module 1250 meeting a preset training end condition.

[0266] In an optional example, as shown in Figure 13 The training module 1250 includes:

[0267] The eighth determination submodule 12501 is configured to determine a current instance prediction information sequence of the current training frame based on the video instance segmentation result of the current training frame. The current instance prediction information sequence includes multiple current instance prediction information, and the multiple current instance prediction information corresponds to the multiple instance candidate boxes.

[0268] The assignment submodule 12503 is configured to assign a tracking identifier to the current instance prediction information sequence based on the video instance segmentation result of the current training frame generated by the second generation module 1240, the annotation information of the predetermined training frame, and historical tracking identifier assignment information of the historical instance prediction information sequence of the predetermined training frame.

[0269] The division submodule 12505 is configured to divide the multiple instance candidate boxes corresponding to the multiple current instance prediction information into a positive sample box set and a negative sample box set based on the current tracking identifier assignment information of the current instance prediction information sequence obtained by the assignment submodule 12503.

[0270] The ninth determination submodule 12507 is configured to determine a model loss value based on the positive sample box set and the negative sample box set obtained by the division submodule 12505.

[0271] The training submodule 12509 is configured to train the initial instance segmentation model based on the model loss value determined by the ninth determination submodule 12507.

[0272] In an optional example, the assigning submodule 12503 includes:

[0273] The first processing unit is configured to, in response to determining, based on the historical tracking identifier assignment information, that the first historical instance prediction information corresponding to the first current instance prediction information in the current instance prediction information sequence determined by the eighth determination submodule 12501 is not assigned a tracking identifier, add the first current instance prediction information to the new instance prediction information set; wherein the first current instance prediction information and the first historical instance prediction information correspond to the fact that the relative position of the first historical instance prediction information in the historical instance prediction information sequence is consistent with the relative position of the first current instance prediction information in the current instance prediction information sequence.

[0274] The second processing unit is configured to, in response to determining, based on the historical tracking identifier assignment information, that the first historical instance prediction information is assigned a first tracking identifier different from the preset tracking identifier, and that the first tracking identifier exists in the annotation information of the current training frame, assign the first tracking identifier to the first current instance prediction information.

[0275] The third processing unit is configured to, in response to determining, based on the historical tracking identifier assignment information, that the first historical instance prediction information is assigned the first tracking identifier, and that the first tracking identifier does not exist in the annotation information of the current training frame, assign the preset tracking identifier to the first current instance prediction information.

[0276] The second determination unit is configured to determine the new instance annotation information set by comparing the annotation information of the current training frame with the annotation information of the predetermined training frame.

[0277] The fourth processing unit is configured to match the new instance annotation information set determined by the second determination unit with the new instance prediction information set, and assign a tracking identifier to at least part of the current instance prediction information in the new instance prediction information set based on the matching result.

[0278] In an optional example, the dividing submodule 12505 includes:

[0279] The third determination unit is configured to determine each current instance prediction information in the current instance prediction information sequence determined by the eighth determination submodule 12501, which is assigned a tracking identifier different from the preset tracking identifier.

[0280] The division unit is configured to divide the set of the respective instance candidate boxes corresponding to the respective current instance prediction information as a positive sample box set, and divide a set of remaining instance candidate boxes as a negative sample box set.

[0281] In an optional example, the ninth determination sub-module 12507 includes:

[0282] The fourth determination unit is configured to determine, from the negative sample box set obtained by the division sub-module 12505, a third instance candidate box with the maximum IoU with the second instance candidate box in the positive sample box set obtained by the division sub-module 12505.

[0283] The fifth determination unit is configured to determine a center distance between the second instance candidate box and the third instance candidate box determined by the fourth determination unit.

[0284] The sixth determination unit is configured to determine a diagonal line length of the second instance candidate box.

[0285] The seventh determination unit is configured to determine the model loss value based on the center distance determined by the fifth determination unit, the diagonal line length determined by the sixth determination unit, a first preset numerical value, and a second preset numerical value.

[0286] In an optional example, the seventh determination unit includes:

[0287] The first operation sub-unit is configured to perform power operation on the center distance determined by the fifth determination unit to obtain a first operation value.

[0288] The second operation sub-unit is configured to perform power operation on the diagonal line length determined by the sixth determination unit to obtain a second operation value.

[0289] The first calculation sub-unit is configured to calculate a ratio of the first operation value obtained by the first operation sub-unit to the second operation value obtained by the second operation sub-unit.

[0290] The second calculation sub-unit is configured to calculate a difference between the first preset numerical value and the ratio obtained by the first calculation sub-unit.

[0291] The selection sub-unit is configured to select a larger value from the difference obtained by the second calculation sub-unit and the second preset numerical value.

[0292] The determination sub-unit is configured to determine the model loss value based on the larger value selected by the selection sub-unit.

[0293] Exemplary electronic device

[0294] Hereinafter, reference will be made to Figure 14An electronic device according to embodiments of the present disclosure is described. The electronic device can be either or both of the first and second devices, or a standalone device independent of them, which can communicate with the first and second devices to receive the captured input signals therefrom.

[0295] Figure 14 A block diagram of an electronic device according to embodiments of the present disclosure is illustrated.

[0296] As Figure 14 illustrated, the electronic device 1400 includes one or more processors 1401 and a memory 1402.

[0297] The processor 1401 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 1400 to perform desired functions.

[0298] The memory 1402 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read-only memory (ROM), hard disk drives, solid-state drives, and / or the like. The computer-readable storage media can store one or more computer program instructions, which the processor 1401 can execute to implement the image generation method of various embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, and the like can also be stored in the computer-readable storage media.

[0299] In one example, the electronic device 1400 can further include an input device 1403 and an output device 1404, which are interconnected through a bus system and / or other form of connection mechanism (not shown).

[0300] For example, when the electronic device is the first or second device, the input device 1403 can be the microphone or microphone array described above. When the electronic device is a standalone device, the input device 1403 can be a communication network connector to receive the captured input signals from the first and second devices.

[0301] In addition, the input device 1403 can further include, for example, a keyboard, a mouse, and the like.

[0302] The output device 1404 can output various information including the determined distance information, direction information, etc. to the outside. The output device 1404 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0303] Of course, in order to simplify, Figure 14 Only some of the components of the electronic device 1400 related to the present disclosure are shown in the middle, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device 1400 can further include any other appropriate components according to specific application cases.

[0304] Exemplary computer program product and computer readable storage medium

[0305] In addition to the above-mentioned methods and devices, embodiments of the present disclosure can also be a computer program product, which includes computer program instructions that, when executed by a processor, cause the processor to perform the steps in the video instance segmentation method according to various embodiments of the present disclosure described in the above "Exemplary Methods" section of the present specification, or the steps in the instance segmentation model training method according to various embodiments of the present disclosure.

[0306] The computer program product can be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0307] In addition, embodiments of the present disclosure can also be a computer readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the video instance segmentation method according to various embodiments of the present disclosure described in the above "Exemplary Methods" section of the present specification, or the steps in the instance segmentation model training method according to various embodiments of the present disclosure.

[0308] The computer readable storage medium can be embodied as one or more combinations of a readable medium and a readable medium can be a readable signal medium or a readable storage medium. A readable storage medium, for example, can include one or more of an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0309] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0310] The block diagrams of the devices, apparatuses, systems, etc. involved in the present disclosure are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. shown in the block diagrams are required or implied. As will be recognized by those skilled in the art, the devices, apparatuses, systems, etc. can be connected, arranged, configured, etc. in any manner and one of ordinary skill can readily appreciate many alternative ways and / or methods for connecting, arranging, configuring, etc. the same. As used in the specification and the claims, the phrase "at least one", in reference to a list of one or more elements, means that at least one element is present and can include one or more of the elements of the list; multiple instances of the same element are not excluded by "at least one". As used herein, the term "another" means at least a second or more. As used herein, the term "wherein" means "wherein the" and "wherein the" means "wherein the". As used herein, the term "and / or" means "and" or "or". As used herein, the term "such as" means "such as but not limited to".

[0311] It should also be noted that in the apparatuses, devices and methods of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present disclosure.

[0312] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to accord the widest scope consistent with the principles and novel features disclosed herein.

[0313] The foregoing description has been presented for the purposes of illustration and description. Furthermore, the description is not intended to limit the embodiments of the disclosure to the forms disclosed herein. Although the various example aspects and embodiments have been described herein with regard to particular aspects and embodiments, those skilled in the art will recognize that certain modifications, changes, substitutions, additions and sub-combinations can be made without departing from the spirit of the disclosure.

Claims

1. A video instance segmentation method, comprising: Feature extraction is performed on the current frame of the video to be processed to obtain the current feature map of the current frame; A reference instance feature sequence is determined based on the historical instance feature sequence corresponding to at least one historical frame in the video to be processed that is located before the current frame. Based on the current feature map, the reference instance feature sequence, and the historical instance candidate box information sequence corresponding to the predetermined frame in the at least one historical frame, the current instance feature sequence of the current frame is determined; wherein, the historical instance candidate box information sequence includes: candidate box information corresponding to each of the multiple historical instance candidate boxes in the historical feature map of the predetermined frame; Based on the current feature map and the current instance feature sequence, the video instance segmentation result of the current frame is generated through the instance segmentation model.

2. The method according to claim 1, wherein, The step of determining a reference instance feature sequence based on the historical instance feature sequences corresponding to at least one historical frame in the video to be processed that is preceding the current frame includes: Determine the first similarity between at least one historical instance feature in the historical instance feature set and the first historical instance feature to obtain at least one first similarity; wherein, the first historical instance feature is a historical instance feature in the historical instance feature sequence corresponding to the predetermined frame, and the historical instance feature set includes: the first historical instance feature, and a historical instance feature in the historical instance feature sequence corresponding to each non-predetermined frame in the at least one historical frame, and any two historical instance features in the historical instance feature set are in the same relative position in their respective historical instance feature sequences; Based on the historical instance feature set and the at least one first similarity, the first reference instance features are determined; Based on the first reference instance feature, a reference instance feature sequence is determined, wherein the relative position of the first reference instance feature in the reference instance feature sequence is consistent with the relative position of the first historical instance feature in the historical instance feature sequence corresponding to the predetermined frame.

3. The method according to claim 2, wherein, The step of determining the first reference instance features based on the historical instance feature set and the at least one first similarity includes: Based on the at least one first similarity, a first weight is assigned to at least one historical instance feature in the historical instance feature set to obtain at least one first weight; Using the at least one first weight, at least one historical instance feature in the historical instance feature set is weighted to obtain a first weighted instance feature; The first weighted instance features and the first historical instance features are fused to obtain the first reference instance features.

4. The method according to claim 1, wherein, Determining the current instance feature sequence of the current frame based on the current feature map, the reference instance feature sequence, and the historical instance candidate box information sequence corresponding to a predetermined frame in at least one historical frame includes: Based on the sequence of historical instance candidate box information, multiple instance candidate boxes are determined in the current feature map; Determine the pixel features of each pixel in the first instance candidate box that corresponds to the second reference instance feature in the reference instance feature sequence within the plurality of instance candidate boxes, thereby obtaining a plurality of pixel features; Based on the second reference instance features and the plurality of pixel features, the first current instance features are determined; Based on the first current instance feature, a current instance feature sequence of the current frame is determined, wherein the relative position of the first current instance feature in the current instance feature sequence is consistent with the relative position of the second reference instance feature in the reference instance feature sequence.

5. The method according to claim 4, wherein, Determining the first current instance feature based on the second reference instance feature and the plurality of pixel features includes: The second similarity between the plurality of pixel features and the second reference instance features is determined respectively, resulting in a plurality of second similarities; Based on the multiple second similarities, second weights are assigned to the multiple pixels respectively to obtain multiple second weights; The multiple second weights are used to weight the multiple pixel features to obtain the first current instance feature.

6. A method for training an instance segmentation model, comprising: Feature extraction is performed on the current training frame in the training video to obtain the current training feature map of the current training frame; Based on the feature sequences of historical training instances corresponding to at least one historical training frame in the training video that is preceding the current training frame, a reference training instance feature sequence is determined. Based on the current training feature map, the reference training instance feature sequence, and the historical training instance candidate box information sequence corresponding to the predetermined training frame in the at least one historical training frame, the current training instance feature sequence of the current training frame is determined; wherein, the historical training instance candidate box information sequence includes: candidate box information corresponding to each of the multiple historical instance candidate boxes in the historical training feature map of the predetermined training frame. Based on the current training feature map and the current training instance feature sequence, the video instance segmentation result of the current training frame is generated through the initial instance segmentation model; The initial instance segmentation model is trained based on the video instance segmentation result of the current training frame, the annotation information of the current training frame, and the annotation information of the predetermined training frame. In response to the initial instance segmentation model after training meeting the preset training termination condition, the initial instance segmentation model after training is determined as the instance segmentation model.

7. The method according to claim 6, wherein, The training of the initial instance segmentation model based on the video instance segmentation result of the current training frame, the annotation information of the current training frame, and the annotation information of the predetermined training frames includes: Based on the video instance segmentation result of the current training frame, a current instance prediction information sequence of the current training frame is determined; wherein, the current instance prediction information sequence includes: multiple current instance prediction information, and the multiple current instance prediction information corresponds to multiple instance candidate boxes; Based on the annotation information of the current training frame, the annotation information of the predetermined training frame, and the historical tracking identifier allocation information of the historical instance prediction information sequence of the predetermined training frame, a tracking identifier is allocated for the current instance prediction information sequence. Based on the current tracking identifier allocation information of the current instance prediction information sequence, the multiple instance candidate boxes corresponding to the multiple current instance prediction information are divided into a positive sample box set and a negative sample box set. Based on the set of positive and negative bounding boxes, the model loss value is determined; The initial instance segmentation model is trained based on the model loss value.

8. The method according to claim 7, wherein, The process of allocating tracking identifiers for the current instance prediction information sequence based on the annotation information of the current training frame, the annotation information of the predetermined training frame, and the historical tracking identifier allocation information of the historical instance prediction information sequence of the predetermined training frame includes: In response to determining, based on the historical tracking identifier allocation information, that the first historical instance prediction information corresponding to the first current instance prediction information in the current instance prediction information sequence has not been assigned a tracking identifier, the first current instance prediction information is added to the new instance prediction information set; wherein, the correspondence between the first current instance prediction information and the first historical instance prediction information indicates that the relative position of the first historical instance prediction information in the historical instance prediction information sequence is consistent with the relative position of the first current instance prediction information in the current instance prediction information sequence; In response to determining, based on the historical tracking identifier allocation information, that the first historical instance prediction information has been assigned a first tracking identifier that is different from the preset tracking identifier, and that the first tracking identifier exists in the annotation information of the current training frame, the first tracking identifier is assigned to the first current instance prediction information. In response to determining, based on the historical tracking identifier allocation information, that the first historical instance prediction information has been assigned the first tracking identifier, and that the first tracking identifier does not exist in the annotation information of the current training frame, the preset tracking identifier is assigned to the first current instance prediction information; By comparing the annotation information of the current training frame with the annotation information of the predetermined training frame, a set of new instance annotation information is determined. The newly added instance annotation information set is matched with the newly added instance prediction information set, and based on the matching result, tracking identifiers are assigned to at least a portion of the current instance prediction information in the newly added instance prediction information set.

9. The method according to claim 8, wherein, The current tracking identifier allocation information based on the current instance prediction information sequence divides the multiple instance candidate boxes corresponding to the multiple current instance prediction information into a positive sample box set and a negative sample box set, including: In the current instance prediction information sequence, determine each current instance prediction information that is assigned a tracking identifier different from the preset tracking identifier; The set of candidate boxes corresponding to each of the current instance prediction information is taken as the positive sample box set, and the set of the remaining candidate boxes is taken as the negative sample box set.

10. The method according to claim 7, wherein, The step of determining the model loss value based on the set of positive sample boxes and the set of negative sample boxes includes: From the set of negative sample boxes, determine the third instance candidate box that has the largest intersection-union ratio with the second instance candidate box in the set of positive sample boxes; Determine the center distance between the second instance candidate box and the third instance candidate box; Determine the diagonal length of the second instance candidate box; The model loss value is determined based on the center distance, the diagonal length, the first preset value, and the second preset value.

11. The method according to claim 10, wherein, The determination of the model loss value based on the center distance, the diagonal length, the first preset value, and the second preset value includes: The center distance is exponentially calculated to obtain the first calculated value; The diagonal length is exponentially calculated to obtain the second calculated value; Calculate the ratio of the first calculated value to the second calculated value; Calculate the difference between the first preset value and the ratio; Select the larger value from the difference and the second preset value; Based on the larger value, the model loss value is determined.

12. A video instance segmentation apparatus, comprising: The first feature extraction module is used to extract features from the current frame of the video to be processed, and obtain the current feature map of the current frame; The first determining module is used to determine a reference instance feature sequence based on the historical instance feature sequence corresponding to at least one historical frame in the video to be processed that is located before the current frame. The second determining module is used to determine the current instance feature sequence of the current frame based on the current feature map obtained by the first feature extraction module, the reference instance feature sequence determined by the first determining module, and the historical instance candidate box information sequence corresponding to the predetermined frame in the at least one historical frame; wherein, the historical instance candidate box information sequence includes: candidate box information corresponding to each of the multiple historical instance candidate boxes in the historical feature map of the predetermined frame. The first generation module is used to generate a video instance segmentation result for the current frame based on the current feature map obtained by the first feature extraction module and the current instance feature sequence determined by the second determination module, via an instance segmentation model.

13. An instance segmentation model training device, comprising: The second feature extraction module is used to extract features from the current training frame in the training video to obtain the current training feature map of the current training frame. The third determining module is used to determine a reference training instance feature sequence based on the feature sequence of each historical training instance corresponding to at least one historical training frame in the training video that is located before the current training frame. The fourth determining module is used to determine the current training instance feature sequence of the current training frame based on the current training feature map obtained by the second feature extraction module, the reference training instance feature sequence determined by the third determining module, and the historical training instance candidate box information sequence corresponding to the predetermined training frame in the at least one historical training frame; wherein, the historical training instance candidate box information sequence includes: candidate box information corresponding to each of the multiple historical instance candidate boxes in the historical training feature map of the predetermined training frame; The second generation module is used to generate the video instance segmentation result of the current training frame based on the current training feature map obtained by the second feature extraction module and the current training instance feature sequence determined by the fourth determination module, via the initial instance segmentation model. The training module is used to train the initial instance segmentation model based on the video instance segmentation result of the current training frame generated by the second generation module, the annotation information of the current training frame, and the annotation information of the predetermined training frame. The fifth determining module is used to determine the trained initial instance segmentation model as an instance segmentation model in response to the training module meeting the preset training termination condition.

14. A computer-readable storage medium storing a computer program for performing the video instance segmentation method according to any one of claims 1-5, or performing the instance segmentation model training method according to any one of claims 6-11.

15. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the video instance segmentation method according to any one of claims 1-5, or to execute the instance segmentation model training method according to any one of claims 6-11.

Images