Semi-supervised training method, device and medium for video semantic segmentation network

Abstract

The present invention discloses a semi-supervised training method, device and medium of a video semantic segmentation network, the method comprising: obtaining a first data set including a plurality of first video frames carrying labels and a second data set including a plurality of second video frames not carrying labels; obtaining a video semantic segmentation network, which comprises a first segmentation network and a second segmentation network; sampling a first video frame from the first data set and then inputting it into the first segmentation network for processing, and determining a first classification loss in combination with the carried labels; sampling a plurality of second video frames from the second data set and then inputting them into the first segmentation network for processing to obtain a plurality of pseudo labels; performing data enhancement on the plurality of second video frames and then inputting them into the second segmentation network for processing, and using a plurality of pseudo labels for supervision to determine a second classification loss; and training the video semantic segmentation network according to the first classification loss and the second classification loss. The present invention can make full use of the time information in the video to improve the semantic segmentation performance.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a semi-supervised training method, device and medium for a video semantic segmentation network. Background Art

[0002] The goal of video semantic segmentation is to predict the pixel-level semantics of each video frame. However, due to the homogeneity of video content, that is, the contents of consecutive different video frames contained in a video are similar in most cases, and the information increment is not much. When using the existing semi-supervised image segmentation technology to complete the video semantic segmentation task, it can only identify consecutive different video frames as a group of independent and unrelated images, and cannot fully utilize the temporal information contained in the video to improve the semantic segmentation performance. Summary of the invention

[0003] The present invention provides a semi-supervised training method, device and medium for a video semantic segmentation network to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0004] In a first aspect, a semi-supervised training method for a video semantic segmentation network is provided, the method comprising:

[0005] Acquire a first data set and a second data set, wherein the first data set includes a plurality of continuous first video frames carrying labels, and the second data set includes a plurality of continuous second video frames not carrying labels;

[0006] Acquire a video semantic segmentation network, which includes a first segmentation network composed of a first feature extractor and a classifier and a second segmentation network composed of a second feature extractor and a time-dependent classifier;

[0007] Randomly sampling a first video frame from the first data set and inputting the first video frame into the first segmentation network for processing, and determining a first classification loss in combination with a label carried by the first video frame;

[0008] Randomly sampling a plurality of continuous second video frames from the second data set, inputting the plurality of second video frames into the first segmentation network for processing, and obtaining a plurality of corresponding pseudo labels;

[0009] After data enhancement, the plurality of second video frames are input into the second segmentation network for processing, and the plurality of pseudo labels are used for supervision to determine a second classification loss;

[0010] The video semantic segmentation network is trained according to the first classification loss and the second classification loss.

[0011] Furthermore, a global prototype and a context prototype are set in the time-related classifier, the global prototype includes learnable weight features corresponding to each category allowed to be detected by the second segmentation network, and the context prototype includes feature means corresponding to each category to which the last video frame input into the second segmentation network belongs.

[0012] Furthermore, in the second segmentation network, the second feature extractor is used to process the input video frame to obtain a feature map, and the time-related classifier is used to determine the predicted probability of each pixel contained in the feature map belonging to each category contained in the global prototype and the predicted probability of each pixel belonging to each category contained in the context prototype based on the similarity between the feature value of each pixel contained in the feature map and each learnable weight feature contained in the global prototype and the similarity between the feature means contained in the context prototype, thereby determining the predicted label corresponding to each pixel as the category with the largest predicted probability.

[0013] Furthermore, the step of performing data enhancement on the plurality of second video frames and then inputting the plurality of second video frames into the second segmentation network for processing, and using the plurality of pseudo labels for supervision to determine the second classification loss comprises:

[0014] Adjusting the color attributes of the plurality of second video frames to obtain a corresponding plurality of third video frames;

[0015] Inputting the plurality of third video frames into the second segmentation network for processing in a manner of increasing timestamps, and using the plurality of pseudo labels for supervision to determine a forward classification loss;

[0016] Inputting the plurality of third video frames into the second segmentation network for processing in a manner of decreasing timestamps, and using the plurality of pseudo labels for supervision to determine a reverse classification loss;

[0017] The forward classification loss is added to the reverse classification loss to obtain a second classification loss.

[0018] Furthermore, the step of inputting the plurality of third video frames into the second segmentation network for processing in a manner of increasing timestamps, and using the plurality of pseudo labels for supervision, and determining the forward classification loss comprises:

[0019] For every two adjacent third video frames in the plurality of third video frames, the third video frame with the larger timestamp is recorded as the first current video frame, and the third video frame with the smaller timestamp is recorded as the first associated video frame; the first associated video frame is input into the second feature extractor for processing to obtain a first feature map, and then combined with the pseudo label corresponding to the first associated video frame, the context prototype currently required by the time-related classifier is determined; the first current video frame is input into the currently updated second segmentation network for processing to obtain a first predicted label;

[0020] A forward classification loss is determined according to a plurality of first prediction labels corresponding to the plurality of third video frames and a plurality of global prototypes and a plurality of context prototypes correspondingly relied upon when processing the plurality of third video frames.

[0021] Furthermore, the step of inputting the plurality of third video frames into the second segmentation network for processing in a manner of decreasing timestamps, and using the plurality of pseudo labels for supervision, and determining the reverse classification loss comprises:

[0022] For every two adjacent third video frames in the plurality of third video frames, the third video frame with a smaller timestamp is recorded as a second current video frame, and the third video frame with a larger timestamp is recorded as a second associated video frame; the second associated video frame is input into the second feature extractor for processing to obtain a second feature map, and then combined with the pseudo-label corresponding to the second associated video frame, the context prototype currently required by the time-related classifier is determined; the second current video frame is input into the currently updated second segmentation network for processing to obtain a second predicted label;

[0023] The reverse classification loss is determined according to the multiple second prediction labels corresponding to the multiple third video frames and the multiple global prototypes and the multiple context prototypes correspondingly relied upon when processing the multiple third video frames.

[0024] Furthermore, the first feature extractor and the second feature extractor have the same network architecture and share weights.

[0025] Furthermore, the method further comprises:

[0026] Acquire multiple continuous video frames to be tested;

[0027] The multiple video frames to be tested are input into the trained second segmentation network in an increasing manner of timestamps for processing to obtain corresponding multiple prediction labels.

[0028] In a second aspect, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the semi-supervised training method of the video semantic segmentation network as described in the first aspect.

[0029] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the semi-supervised training method of the video semantic segmentation network as described in the first aspect is implemented.

[0030] The present invention has at least the following beneficial effects: by introducing contextual prototypes and global prototypes into a video semantic segmentation network, the content correlation between consecutive different video frames can be effectively utilized, and temporal consistency constraints across frames are implicitly imposed during the unsupervised training process of the video semantic segmentation network. The semantic concepts of historical video frames are used to assist in identifying the semantic concepts of the current video frame, and forward classification loss and reverse classification loss are introduced in the training process of the time-related classifier, thereby improving the video semantic segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0032] Figure 1 It is a flowchart of a semi-supervised training method of a video semantic segmentation network in an embodiment of the present invention;

[0033] Figure 2 It is a schematic diagram of the hardware structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0037] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0038] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all content and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0039] Please refer to Figure 1 , Figure 1 1 is a flow chart of a semi-supervised training method for a video semantic segmentation network provided by an embodiment of the present invention, wherein the method comprises the following steps:

[0040] Step S110, obtaining a first data set and a second data set, wherein the first data set includes a plurality of continuous first video frames carrying labels, and the second data set includes a plurality of continuous second video frames not carrying labels;

[0041] Step S120, obtaining a video semantic segmentation network, which includes a first segmentation network composed of a first feature extractor and a classifier and a second segmentation network composed of a second feature extractor and a time-dependent classifier;

[0042] Step S130: randomly sample a first video frame from the first data set and input the first video frame into the first segmentation network for processing, and determine a first classification loss in combination with a label carried by the first video frame;

[0043] Step S140: randomly sampling a plurality of continuous second video frames from the second data set, inputting the plurality of second video frames into the first segmentation network for processing, and obtaining a plurality of corresponding pseudo labels;

[0044] Step S150, performing data enhancement on the plurality of second video frames and then inputting them into the second segmentation network for processing, and using the plurality of pseudo labels for supervision to determine a second classification loss;

[0045] Step S160: training the video semantic segmentation network according to the first classification loss and the second classification loss.

[0046] In the embodiment of the present invention, the various network components mentioned in the above step S120 are described as follows:

[0047] Firstly, the network architecture of the first feature extractor and the second feature extractor is the same, preferably an existing ResNet network (i.e., a deep residual network), and the two feature extractors share weights during training;

[0048] Second, the classifier still uses the traditional classifier, which mainly includes a 1×1 full convolution layer and a softmax classification layer connected in sequence;

[0049] Thirdly, the time-dependent classifier is provided with a context prototype and a global prototype, wherein:

[0050] The global prototype includes learnable weight features corresponding to each category that can be detected by the second segmentation network, denoted as C is the total number of categories that can be detected by the second segmentation network, n is the feature dimension output by the second feature extractor, is the learnable weight feature corresponding to the i-th category; it should be noted that the global prototype remains unchanged during the forward reasoning process, but needs to be updated during the back-propagation training process;

[0051] The context prototype includes the feature mean corresponding to each category of the last video frame input to the second segmentation network, denoted as K is the total number of categories to which the last video frame input to the second segmentation network belongs, and the K categories involved in the context prototype are actually included in the C categories involved in the global prototype, that is, K≤C, n is the feature dimension output by the second feature extractor, is the feature mean corresponding to the jth category.

[0052] On this basis, the internal implementation process of the second segmentation network mainly proposed in the present invention is described, which specifically includes the following steps:

[0053] (1) When the second segmentation network receives an input video frame, the second feature extractor processes the input video frame to obtain a feature map;

[0054] (2) Determine, by means of the time-dependent classifier, the predicted probability that each pixel contained in the feature map belongs to each category contained in the context prototype, and the predicted probability that each pixel contained in the feature map belongs to each category contained in the global prototype based on the similarity between the feature value of each pixel contained in the feature map and the mean value of each feature contained in the context prototype, and the similarity between the feature value of each pixel contained in the feature map and each learnable weight feature contained in the global prototype, and further determine that the predicted label corresponding to each pixel is the category with the largest predicted probability;

[0055] Among them, the predicted probability that the k-th pixel contained in the feature map belongs to the i-th category contained in the global prototype can be calculated by the following mathematical expression:

[0056]

[0057] The predicted probability that the k-th pixel contained in the feature map belongs to the j-th category contained in the context prototype can be calculated by the following mathematical expression:

[0058]

[0059] In the formula, is the predicted probability that the k-th pixel contained in the feature map belongs to the i-th category contained in the global prototype, is the predicted probability that the k-th pixel contained in the feature map belongs to the j-th category contained in the context prototype, η G is the control parameter set for the global prototype, η C is the control parameter set for the context prototype, η G and η C Mainly used to control the peak distribution of the softmax function. is the learnable weight feature corresponding to the i-th category contained in the global prototype The normalized result is is the feature mean corresponding to the jth category contained in the context prototype The normalized result is is the feature value of the kth pixel contained in the feature map The normalized result is<a,b> Represents the cosine similarity value between feature a and feature b and the corresponding value range is [-1,1], (a,b) is or e represents the exponential function.

[0060] It should be noted that the update mechanisms of the global prototype and the context prototype are different and the features of the new input are similar to the context prototype. This may cause the global prototype to be ignored in the early stage of network training, resulting in the network only being able to learn the similarities between frames but not the semantic concepts on the video frames that do not carry labels. Therefore, before feature classification, it is necessary to normalize the new input features, the various learnable weight features contained in the global prototype, and the mean of the various features contained in the context prototype to narrow the distribution differences between them and achieve a fairer network learning process.

[0061] In the embodiment of the present invention, the implementation process of the above step S130 includes but is not limited to the following:

[0062] Step S131: randomly sampling the first data set to obtain a first video frame;

[0063] Step S132: When the first segmentation network receives the first video frame, the first feature extractor processes the first video frame to obtain a first feature map, and the classifier processes the first feature map to obtain a first pseudo label;

[0064] Step S133: Calculate the difference between the first pseudo label and the label carried by the first video frame using the existing cross entropy loss function, and then define the difference result as the first classification loss.

[0065] In the embodiment of the present invention, the implementation process of the above step S150 includes but is not limited to the following:

[0066] Step S151, adjusting the color attributes of the plurality of second video frames to obtain a plurality of corresponding third video frames; wherein the color attribute adjustment method is to randomly change the brightness, randomly change the contrast, or randomly change the saturation;

[0067] Step S152: input the plurality of third video frames into the second segmentation network in sequence for processing in an ascending manner of timestamps, and use the plurality of pseudo labels as supervisory signals during the processing to determine a forward classification loss;

[0068] Step S153: input the plurality of third video frames into the second segmentation network in sequence for processing in a decreasing manner of timestamps, and use the plurality of pseudo labels as supervisory signals during the processing to determine the reverse classification loss;

[0069] Step S154: Add the reverse classification loss and the forward classification loss to obtain a second classification loss.

[0070] It should be noted that, during the network training process, the above-mentioned step S152 and the above-mentioned step S153 are used to predict and classify the current video frame, while taking into account the feature information of the previous video frame and the feature information of the next video frame, which can optimize and stabilize the training and enable better synergy between the global prototype and the context prototype in the semantic classification process.

[0071] More specifically, the implementation process of the above step S152 includes but is not limited to the following:

[0072] Step S152.1, obtaining every two third video frames that are adjacent to each other in the plurality of third video frames, defining the third video frame with a larger timestamp as a first current video frame, and defining the third video frame with a smaller timestamp as a first associated video frame;

[0073] Step S152.2: input the first associated video frame into the second feature extractor for processing to obtain a first feature map;

[0074] Step S152.3, determining the context prototype to which the time-related classifier currently needs to be applied according to the pseudo-label corresponding to the first associated video frame and the first feature map;

[0075] Step S152.4: input the first current video frame into the currently updated second segmentation network for processing to obtain a first predicted label;

[0076] When processing the third video frame with the smallest timestamp, the time-related classifier included in the second segmentation network is only applied to the global prototype for classification prediction; then the other third video frames are processed according to the above steps S152.1 to S152.4 to obtain multiple first prediction labels corresponding to the multiple third video frames;

[0077] Step S152.5, combining the multiple context prototypes and the multiple global prototypes that the multiple third video frames depend on during the processing and the multiple first prediction labels corresponding to the multiple third video frames, calculate the forward classification loss through the following mathematical expression:

[0078]

[0079] Where, L TC is the forward classification loss, T is the number of the plurality of third video frames, H is the height of each third video frame, W is the width of each third video frame, is the predicted probability that the k-th pixel contained in the t-th third video frame belongs to the i-th category contained in the global prototype that the t-th third video frame relies on during the processing, is the predicted probability that the kth pixel contained in the tth third video frame belongs to the jth category contained in the context prototype (that is, the context prototype determined by the t-1th third video frame) on which the tth third video frame depends during processing, and it is limited that the jth category contained in the context prototype is the same as the ith category contained in the global prototype, and both are the first predicted label corresponding to the kth pixel contained in the tth third video frame, χ(·) represents the assignment function, and when the jth category contained in the context prototype exists, set When the jth category contained in the context prototype does not exist, set

[0080] It should be noted that the above-mentioned step S152.5 is proposed considering that when the first predicted label corresponding to any pixel contained in the first feature map exists in both the global prototype and the context prototype on which the first feature map depends during the processing, it should not be penalized by the loss function; illustratively, assuming that the first predicted label corresponding to a pixel contained in the first feature map is A, the global prototype on which the first feature map depends during the processing records the learnable weight features corresponding to category A, and the context prototype on which the first feature map depends during the processing also records the feature mean corresponding to category A. Since the global prototype and the context prototype only make different semantic descriptions of category A, no matter whether the feature value of the pixel is more similar to the global prototype or the feature value of the pixel is more similar to the context prototype, it should not be penalized by the loss function.

[0081] More specifically, the implementation process of step S153 includes but is not limited to the following:

[0082] Step S153.1, obtaining every two third video frames that are adjacent to each other in the plurality of third video frames, defining the third video frame with a smaller timestamp as a second current video frame, and defining the third video frame with a larger timestamp as a second associated video frame;

[0083] Step S153.2: input the second associated video frame into the second feature extractor for processing to obtain a second feature map;

[0084] Step S153.3, determining the context prototype to which the time-related classifier currently needs to be applied according to the pseudo-label corresponding to the second associated video frame and the second feature map;

[0085] Step S153.4: input the second current video frame into the currently updated second segmentation network for processing to obtain a second predicted label;

[0086] When processing the third video frame with the largest timestamp, the time-related classifier included in the second segmentation network is only applied to the global prototype for classification prediction; then the other third video frames are processed according to the above steps S153.1 to S153.4 to obtain multiple second prediction labels corresponding to the multiple third video frames;

[0087] Step S153.5, combining the multiple context prototypes and the multiple global prototypes that the multiple third video frames depend on during the processing and the multiple second prediction labels corresponding to the multiple third video frames, calculate the reverse classification loss through the following mathematical expression:

[0088]

[0089] Where, L Bi-TC is the reverse classification loss, T is the number of the plurality of third video frames, H is the height of each third video frame, W is the width of each third video frame, is the predicted probability that the k-th pixel contained in the t-th third video frame belongs to the i-th category contained in the global prototype that the t-th third video frame relies on during the processing, is the predicted probability that the kth pixel contained in the tth third video frame belongs to the jth category contained in the context prototype (that is, the context prototype determined by the t+1th third video frame) on which the tth third video frame depends during processing, and it is limited that the jth category contained in the context prototype is the same as the ith category contained in the global prototype, and both are the second predicted labels corresponding to the kth pixel contained in the tth third video frame. X(·) represents the assignment function. When the jth category contained in the context prototype exists, set When the jth category contained in the context prototype does not exist, set

[0090] It should be noted that the above step S153.5 is proposed considering that when the second predicted label corresponding to any pixel contained in the second feature map exists in both the global prototype and the context prototype that the second feature map depends on during the processing, it should not be penalized by the loss function.

[0091] In the above step S152.3 and the above step S153.3, the following mathematical expression may be used to determine the context prototype:

[0092]

[0093] In the formula, for the feature map corresponding to the associated video frame, F k is the feature value of the kth pixel contained in the feature map, represents the exponential function, PLk,j The k-th pixel contained in the feature map belongs to the j-th category pseudo label, that is, the pixel-level features belonging to the same category are grouped according to the pseudo label corresponding to the associated video frame.

[0094] In the embodiment of the present invention, the implementation process of the above step S160 includes but is not limited to the following:

[0095] Step S161: Combining the first classification loss and the second classification loss, the total classification loss required for network training is calculated by the following mathematical expression:

[0096] L=L sup +λL unsup ;

[0097] In the formula, L is the total classification loss, L sup is the first classification loss, L unsup is the second classification loss, λ is the adjustment coefficient;

[0098] Step S162: Based on the total classification loss, the video semantic segmentation network is trained using an existing back propagation algorithm.

[0099] In the embodiment of the present invention, after completing the training task of the video semantic segmentation network through the above steps S110 to S160, the trained second segmentation network can be put into formal application, which specifically includes the following steps:

[0100] (1) Acquire multiple consecutive video frames to be tested;

[0101] (2) Inputting the multiple video frames to be tested into the trained second segmentation network in sequence according to the increasing timestamps for processing to obtain the corresponding multiple prediction labels.

[0102] It should be noted that when processing the test video frame with the smallest timestamp, the time-related classifier contained in the trained second segmentation network is only applied to the global prototype for classification prediction; for every two test video frames that are adjacent to each other in other test video frames, the test video frame with a smaller timestamp is defined as the first test video frame, and the test video frame with a larger timestamp is defined as the second test video frame. When processing the second test video frame, the time-related classifier contained in the trained second segmentation network needs to be applied to the global prototype and the context prototype for classification prediction, and the context prototype is determined based on the feature map and predicted label obtained by the trained second segmentation network when processing the first test video frame, that is, the trained first segmentation network is no longer used to determine the pseudo label to update the context prototype.

[0103] In an embodiment of the present invention, by introducing contextual prototypes and global prototypes into a video semantic segmentation network, the content correlation between consecutive different video frames can be effectively utilized, and temporal consistency constraints across frames can be implicitly imposed during the unsupervised training process of the video semantic segmentation network. The semantic concepts of historical video frames are used to assist in identifying the semantic concepts of the current video frame, and forward classification loss and reverse classification loss are introduced in the training process of the time-related classifier, thereby improving the video semantic segmentation performance.

[0104] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, a semi-supervised training method for a video semantic segmentation network in the above embodiment is implemented. Wherein, the computer-readable storage medium includes but is not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. In other words, the storage device includes any medium that stores or transmits information in a readable form by a device (such as a computer, a mobile phone, etc.), which can be a read-only memory, a disk or an optical disk, etc.

[0105] also, Figure 2 2 is a schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention, wherein the computer device includes a processor 220, a memory 230, an input unit 240, a display unit 250 and other devices. Those skilled in the art can understand that Figure 2The device structure device shown does not constitute a limitation on all devices, and may include more or fewer components than shown, or combine certain components. The memory 230 can be used to store the computer program 210 and various functional modules, and the processor 220 runs the computer program 210 stored in the memory 230, thereby executing various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include an internal memory and an external memory. The internal memory may include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a random access memory. The external memory may include a hard disk, a floppy disk, a U disk, a magnetic tape, etc. The memory 230 disclosed in the embodiment of the present invention includes but is not limited to the above-mentioned types of memory. The memory 230 disclosed in the embodiment of the present invention is only an example and not a limitation.

[0106] The input unit 240 is used to receive the input of the signal and the keyword input by the user. The input unit 240 may include a touch panel and other input devices. The touch panel can collect the user's touch operation on or near it (such as the user's operation on or near the touch panel using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program; other input devices may include but are not limited to one or more of a physical keyboard, a function key (such as a playback control key, a switch key, etc.), a trackball, a mouse, a joystick, etc. The display unit 250 can be used to display the information input by the user or the information provided to the user and various menus of the terminal device. The display unit 250 can be in the form of a liquid crystal display, an organic light emitting diode, etc. The processor 220 is the control center of the terminal device, which uses various interfaces and lines to connect various parts of the entire device, and executes various functions and processes data by running or executing software programs and / or modules stored in the memory 230, and calling data stored in the memory 230.

[0107] As an embodiment, the computer device includes a processor 220, a memory 230 and a computer program 210, wherein the computer program 210 is stored in the memory 230 and is configured to be executed by the processor 220, and the computer program 210 is configured to execute a semi-supervised training method for a video semantic segmentation network in the above-mentioned embodiment.

[0108] The terms "including" and "having" and any variations thereof in the specification of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0109] In the present application, it should be understood that "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0110] Although the description of the present application has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims by reference to the attached claims, taking into account the prior art, so as to effectively cover the intended scope of the present application. In addition, the above description of the present application is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present application that have not yet been foreseen may still represent equivalent changes to the present application.

Claims

1. A semi-supervised training method for a video semantic segmentation network, characterized in that: The method comprises: Acquire a first data set and a second data set, wherein the first data set includes a plurality of continuous first video frames carrying labels, and the second data set includes a plurality of continuous second video frames not carrying labels; Obtain a video semantic segmentation network, which includes a first segmentation network composed of a first feature extractor and a classifier and a second segmentation network composed of a second feature extractor and a time-dependent classifier, wherein a global prototype and a context prototype are set in the time-dependent classifier, the global prototype includes learnable weight features corresponding to each category allowed to be detected by the second segmentation network, and the context prototype includes feature means corresponding to each category to which a video frame previously input into the second segmentation network belongs; Randomly sampling a first video frame from the first data set and inputting the first video frame into the first segmentation network for processing, and determining a first classification loss in combination with a label carried by the first video frame; Randomly sampling a plurality of continuous second video frames from the second data set, inputting the plurality of second video frames into the first segmentation network for processing, and obtaining a plurality of corresponding pseudo labels; After data enhancement, the plurality of second video frames are input into the second segmentation network for processing, and the plurality of pseudo labels are used for supervision to determine a second classification loss; The video semantic segmentation network is trained according to the first classification loss and the second classification loss.

2. The semi-supervised training method for video semantic segmentation network according to claim 1, characterized in that: In the second segmentation network, the second feature extractor is used to process the input video frame to obtain a feature map, and the time-related classifier is used to determine the predicted probability of each pixel contained in the feature map belonging to each category contained in the global prototype and the predicted probability of each pixel belonging to each category contained in the context prototype based on the similarity between the feature value of each pixel contained in the feature map and each learnable weight feature contained in the global prototype and the similarity between the feature means contained in the context prototype, so as to further determine the predicted label corresponding to each pixel as the category with the largest predicted probability.

3. The semi-supervised training method for video semantic segmentation network according to claim 2, characterized in that: The step of performing data enhancement on the plurality of second video frames and then inputting the plurality of second video frames into the second segmentation network for processing, and using the plurality of pseudo labels for supervision to determine the second classification loss comprises: Adjusting the color attributes of the plurality of second video frames to obtain a corresponding plurality of third video frames; Inputting the plurality of third video frames into the second segmentation network for processing in a manner of increasing timestamps, and using the plurality of pseudo labels for supervision to determine a forward classification loss; Inputting the plurality of third video frames into the second segmentation network for processing in a manner of decreasing timestamps, and using the plurality of pseudo labels for supervision to determine a reverse classification loss; The forward classification loss is added to the reverse classification loss to obtain a second classification loss.

4. The semi-supervised training method for a video semantic segmentation network according to claim 3, characterized in that: The step of inputting the plurality of third video frames into the second segmentation network for processing in a manner of increasing timestamps, and using the plurality of pseudo labels for supervision to determine the forward classification loss comprises: For every two adjacent third video frames in the plurality of third video frames, the third video frame with the larger timestamp is recorded as the first current video frame, and the third video frame with the smaller timestamp is recorded as the first associated video frame; the first associated video frame is input into the second feature extractor for processing to obtain a first feature map, and then combined with the pseudo label corresponding to the first associated video frame, the context prototype currently required by the time-related classifier is determined; the first current video frame is input into the currently updated second segmentation network for processing to obtain a first predicted label; A forward classification loss is determined according to a plurality of first prediction labels corresponding to the plurality of third video frames and a plurality of global prototypes and a plurality of context prototypes correspondingly relied upon when processing the plurality of third video frames.

5. The semi-supervised training method for a video semantic segmentation network according to claim 3, characterized in that: The step of inputting the plurality of third video frames into the second segmentation network for processing in a manner of decreasing timestamps, and using the plurality of pseudo labels for supervision to determine the reverse classification loss comprises: For every two adjacent third video frames in the plurality of third video frames, the third video frame with a smaller timestamp is recorded as a second current video frame, and the third video frame with a larger timestamp is recorded as a second associated video frame; the second associated video frame is input into the second feature extractor for processing to obtain a second feature map, and then combined with the pseudo-label corresponding to the second associated video frame, the context prototype currently required by the time-related classifier is determined; the second current video frame is input into the currently updated second segmentation network for processing to obtain a second predicted label; The reverse classification loss is determined according to the multiple second prediction labels corresponding to the multiple third video frames and the multiple global prototypes and the multiple context prototypes correspondingly relied upon when processing the multiple third video frames.

6. The semi-supervised training method for a video semantic segmentation network according to claim 1, characterized in that: The first feature extractor and the second feature extractor have the same network architecture and share weights.

7. The semi-supervised training method for a video semantic segmentation network according to claim 1, characterized in that: The method further comprises: Acquire multiple continuous video frames to be tested; The multiple video frames to be tested are input into the trained second segmentation network in an increasing manner of timestamps for processing to obtain corresponding multiple prediction labels.

8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: The processor executes the computer program to implement the semi-supervised training method for the video semantic segmentation network as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the semi-supervised training method for a video semantic segmentation network according to any one of claims 1 to 7 is implemented.

Images