A method, device, apparatus and readable storage medium for semantic segmentation

CN114187451BActive Publication Date: 2024-11-05GUANGZHOU WERIDE TECH LTD CO +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111556147.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-17
Publication Date
2024-11-05
Estimated Expiration
2041-12-17

Smart Images

  • Figure CN114187451B_ABST
    Figure CN114187451B_ABST
Patent Text Reader

Abstract

The present application discloses a method, apparatus, device and readable storage medium for semantic segmentation, which obtains the current frame image, the previous frame image and the previous frame annotated image of the video to be annotated, and inputs the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain the current frame annotated image, wherein the semantic segmentation model uses the current frame training image, the previous frame training image and the previous frame training annotated image as training samples, and is trained with the types of each pixel annotated in the current frame training image as training labels. In the present application, when semantic segmentation is performed on the current frame image, the previous frame image and the previous frame annotated image are input into the semantic segmentation model together with the current frame image. When the difference between adjacent frame images in the video to be annotated is small, the semantic segmentation model can combine the previous frame image and the previous frame annotated image to annotate the type of each pixel in the current frame image, thereby improving the accuracy of the annotation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method, apparatus, device and readable storage medium for semantic segmentation. Background Art

[0002] Semantic segmentation is a basic task in the field of computer vision, and its purpose is to label each pixel in an image with a semantic label. Before the widespread use of neural network models, semantic segmentation was mostly achieved based on traditional methods such as random forest classifiers. With the continuous development of science and technology, neural network models have begun to be applied to various fields, and people have also begun to use neural network models for semantic segmentation. For example, in the field of autonomous driving, a trained semantic segmentation model can be used to label the type of each pixel in the road image collected by the autonomous driving vehicle. However, when autonomous driving vehicles collect road information, they collect it more through videos. Therefore, how to semantically segment images in videos has always been a concern for people. Summary of the invention

[0003] In view of this, the present application provides a semantic segmentation method, apparatus, device and readable storage medium to facilitate semantic segmentation of images in a video.

[0004] In order to achieve the above objectives, the proposed solution is as follows:

[0005] A semantic segmentation method, comprising:

[0006] Obtaining a current frame image, a previous frame image, and a previous annotated image of the video to be annotated, wherein the previous annotated image contains the type of each pixel point;

[0007] Input the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain a current frame annotated image;

[0008] The semantic segmentation model uses the current frame training image, the previous frame training image and the previous frame annotated training image as training samples, and is trained using the types of each pixel annotated in the current frame training image as training labels.

[0009] Optionally, the semantic segmentation model includes: an input layer, a current frame image annotation layer, and a previous frame image annotation layer cascaded in sequence;

[0010] The training process of the semantic segmentation model includes:

[0011] Through the input layer, obtain the current frame training image, the previous frame training image and the previous frame annotated training image;

[0012] Through the current frame image annotation layer, based on the previous frame training image and the previous frame annotated training image, the current frame training image is annotated, and the current frame annotated image is predicted;

[0013] The previous frame of training image is annotated through the previous frame image annotation layer, and the previous frame of annotated image is predicted;

[0014] The parameters of the semantic segmentation model are updated with the predicted type of each pixel in the current frame annotated image being close to the type label of each pixel in the current frame training image as the training goal, and the predicted type of each pixel in the previous frame annotated image being close to the type label of each pixel in the previous frame annotated training image as the training goal.

[0015] Optionally, inputting the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain the current frame annotated image includes:

[0016] Inputting the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model;

[0017] The current frame image is annotated using the current frame image annotation layer of the semantic segmentation model based on the previous frame image and the previous frame annotated image to output the current frame annotated image.

[0018] Optionally, also include:

[0019] The current frame image is used as the previous frame image, the current frame annotated image is used as the previous frame annotated image, and the next frame image in the video to be annotated is used as the current frame image, and the step of inputting the current frame image, the previous frame image, and the previous annotated image into the semantic segmentation model is returned until all frame images in the video to be annotated are annotated images.

[0020] Optionally, also include:

[0021] Record the number of labeled images in the video to be labeled;

[0022] Determining whether the number is a multiple of a preset threshold;

[0023] If the number is a multiple of the preset threshold, receiving verification information sent by the user, and judging whether the user has any objection to the annotated image of the current frame based on the verification information;

[0024] If the user has any objection to the current frame annotated image, the current frame annotated image modified by the user is obtained and replaced with the current frame annotated image output by the semantic segmentation model in the current prediction cycle.

[0025] A semantic segmentation device, comprising:

[0026] An image acquisition unit, used to acquire a current frame image, a previous frame image and a previous annotated image of the video to be annotated, wherein the previous annotated image includes the type of each pixel point;

[0027] An image annotation unit, used for inputting the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain a current frame annotated image;

[0028] The semantic segmentation model uses the current frame training image, the previous frame training image and the previous frame annotated training image as training samples, and is trained using the types of each pixel annotated in the current frame training image as training labels.

[0029] Optionally, also include:

[0030] A model training unit, used for training to obtain the semantic segmentation model, wherein the semantic segmentation model comprises: an input layer, a current frame image annotation layer, and a previous frame image annotation layer cascaded in sequence;

[0031] The training process of the semantic segmentation model includes:

[0032] Through the input layer, obtain the current frame training image, the previous frame training image and the previous frame annotated training image;

[0033] Through the current frame image annotation layer, based on the previous frame training image and the previous frame annotated training image, the current frame training image is annotated, and the current frame annotated image is predicted;

[0034] The previous frame of training image is annotated through the previous frame image annotation layer, and the previous frame of annotated image is predicted;

[0035] The parameters of the semantic segmentation model are updated with the predicted type of each pixel in the current frame annotated image being close to the type label of each pixel in the current frame training image as the training goal, and the predicted type of each pixel in the previous frame annotated image being close to the type label of each pixel in the previous frame annotated training image as the training goal.

[0036] Optionally, also include:

[0037] A cyclic annotation unit is used to take the current frame image as the previous frame image, the current frame annotated image as the previous frame annotated image, and the next frame image in the video to be annotated as the current frame image, and return to execute the step of inputting the current frame image, the previous frame image and the previous frame annotated image into the semantic segmentation model.

[0038] A semantic segmentation device, comprising: a memory and a processor;

[0039] The memory is used to store programs;

[0040] The processor is used to execute the program to implement the various steps of the semantic segmentation method as described above.

[0041] A readable storage medium stores a computer program, which, when executed by a processor, implements the various steps of the semantic segmentation method as described above.

[0042] It can be seen from the above technical solution that the embodiment of the present application provides a method, device, equipment and readable storage medium for semantic segmentation, by obtaining the current frame image, the previous frame image and the previous frame annotated image containing the type of each pixel point of the video to be annotated, and inputting the current frame image, the previous frame image and the previous frame annotated image into the semantic segmentation model to obtain the current frame annotated image, thereby realizing semantic segmentation of the image in the video to be annotated, wherein the semantic segmentation model uses the current frame training image, the previous frame training image and the previous frame training annotated image as training samples, and is trained with the type of each pixel point annotated in the current frame training image as training labels.

[0043] Furthermore, in the present application, when semantic segmentation is performed on the current frame image, the previous frame image and the previous annotated frame image are also input into the semantic segmentation model together with the current frame image. When the adjacent frame images in the video to be annotated are not much different, the semantic segmentation model can combine the previous frame image and the previous annotated frame image to annotate the type of each pixel in the current frame image, thereby improving the accuracy of the annotation to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 A flowchart of an optional semantic segmentation method provided in an embodiment of the present application;

[0046] Figure 2 An optional architecture diagram of a semantic segmentation model provided in an embodiment of the present application;

[0047] Figure 3 A flowchart of an optional method for training a semantic segmentation model provided in an embodiment of the present application;

[0048] Figure 4A flow chart of a prediction method for a semantic segmentation model provided in an embodiment of the present application;

[0049] Figure 5 A flowchart of an optional semantic segmentation method provided in an embodiment of the present application;

[0050] Figure 6 A flowchart of an optional semantic segmentation method provided in an embodiment of the present application;

[0051] Figure 7 A schematic diagram of the structure of a semantic segmentation device provided in an embodiment of the present application;

[0052] Figure 8 This is a hardware structure block diagram of the semantic segmentation device disclosed in the embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] Figure 1 A flowchart of an optional semantic segmentation method provided in an embodiment of the present application, the method may include the following steps:

[0055] Step S100: Obtain the current frame image, the previous frame image and the previous annotated frame image of the video to be annotated.

[0056] The above-mentioned previous frame of annotated image includes the type of each pixel point.

[0057] Specifically, a certain frame image is determined from the video to be annotated as the current frame image. After the current frame image is determined, the previous frame image adjacent to the current frame can be used as the previous frame image. Each pixel point in the previous annotated frame image is annotated with a corresponding type, such as vehicles, trees, pedestrians, buildings, roads and roadblocks, etc. The previous annotated frame image can be obtained by manually annotating the type of each pixel point in the previous frame image after the previous frame image is determined, thereby obtaining the previous annotated frame image; or after the previous frame image is determined, the previous frame image is processed by a labeling model trained with images of the types of the annotated pixels as training data, thereby obtaining the previous annotated frame image.

[0058] Step S101: input the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain the current frame annotated image.

[0059] The semantic segmentation model is trained by using the current frame training image, the previous frame training image and the previous frame annotated training image as training samples, and the type of each pixel annotated in the current frame training image is used as a training label.

[0060] Specifically, the current frame image, the previous frame image and the previous frame annotated image obtained in the above steps are input into the semantic segmentation model to obtain the current frame annotated image. In this step, the trained semantic segmentation model can be used to perform semantic segmentation on the current frame image, and the input of the semantic segmentation model is the current frame image, the previous frame image and the previous frame annotated image. This makes it possible that when the adjacent frame images in the video to be annotated are not much different, since there are many common points in the current frame image and the previous frame image, the semantic segmentation model can combine the previous frame image and the previous frame annotated image to annotate the type of each pixel in the current frame image, thereby improving the accuracy of the annotation to a certain extent.

[0061] In the above embodiment, a method for semantic segmentation is provided, by obtaining a current frame image, a previous frame image and a previous frame annotated image containing the type of each pixel point of the video to be annotated, and inputting the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain the current frame annotated image, thereby realizing semantic segmentation of the image in the video to be annotated, wherein the semantic segmentation model uses the current frame training image, the previous frame training image and the previous frame training annotated image as training samples, and is trained with the type of each pixel point annotated in the current frame training image as training labels.

[0062] Furthermore, in the present application, when semantic segmentation is performed on the current frame image, the previous frame image and the previous annotated frame image are also input into the semantic segmentation model together with the current frame image. When the adjacent frame images in the video to be annotated are not much different, the semantic segmentation model can combine the previous frame image and the previous annotated frame image to annotate the type of each pixel in the current frame image, thereby improving the accuracy of the annotation to a certain extent.

[0063] In some embodiments of the present application, reference Figure 2 As shown, Figure 2 An optional architecture diagram of a semantic segmentation model provided in an embodiment of the present application, the architecture of the semantic segmentation model may include: an input layer, a current frame image annotation layer, and a previous frame image annotation layer cascaded in sequence.

[0064] Among them, the input layer can obtain the current frame image, the previous frame image and the previous annotated image in the video to be annotated. The current frame image annotation layer can annotate the current frame image based on the previous frame image and the previous annotated image, and predict the current frame annotated image. The previous frame image annotation layer can annotate the previous frame image and predict the previous frame annotated image.

[0065] Combination Figure 2 and Figure 3 As shown, an optional training process of a semantic segmentation model provided in an embodiment of the present application is described. The training process of the semantic segmentation model may include:

[0066] Step S200: Obtain the current frame training image, the previous frame training image and the previous frame labeled training image through the input layer.

[0067] Specifically, after the input layer obtains the current frame training image, the previous frame training image, and the previous frame labeled training image, the current frame training image, the previous frame training image, and the previous frame labeled training image can be vectorized and output to the current frame image labeling layer. In the process of training the semantic segmentation model, the type of each pixel in the current frame training image is known.

[0068] Step S201: annotate the current frame training image based on the previous frame training image and the previous frame annotated training image through the current frame image annotation layer, and predict the current frame annotated image.

[0069] Specifically, the current frame image annotation layer can annotate the current frame training image based on the previous frame training image and the previous frame annotated image output by the input layer, and predict the current frame annotated image. In this step, the current frame image annotation layer of the semantic segmentation model can use the previous frame image and the previous frame annotated image as references when annotating the types of each pixel point in the current frame image. This allows the accuracy of annotation to be improved to a certain extent when annotating the types of each pixel point in the current frame image when the adjacent frame images in the video to be annotated are not much different, because there will be more common points in the current frame image and the previous frame image.

[0070] Step S202: annotate the previous frame of training image through the previous frame image annotation layer, and predict and obtain the previous frame of annotated image.

[0071] Specifically, after obtaining the previous frame of training image output by the input layer in the above step, the previous frame image annotation layer can annotate each pixel in the previous frame of training image, thereby predicting the previous frame of annotated image.

[0072] The execution order of step S201 and step S202 can be performed simultaneously or in any order. No matter which execution order is adopted, it does not affect the implementation of the embodiment of the present application. Figure 3 An alternative execution order is illustrated in FIG.

[0073] Step S203: Update the parameters of the semantic segmentation model according to the pre-set training objectives.

[0074] Specifically, in this step, the parameters of the semantic segmentation model can be updated by taking the predicted type of each pixel in the current frame annotated image as being close to the type label of each pixel in the current frame training image as the training goal, and taking the predicted type of each pixel in the previous frame annotated image as being close to the type label of each pixel in the previous frame annotated training image as the training goal.

[0075] The process of training the semantic segmentation model not only takes the predicted type of each pixel in the current frame annotated image as close to the type label of each pixel in the current frame training image as the training goal, but also adds the predicted type of each pixel in the previous frame annotated image as close to the type label of each pixel in the previous frame annotated training image as the training goal, forming a multi-task training process to update the parameters of the semantic segmentation model.

[0076] In the above embodiment, by adding a training target in which the type of each pixel point in the previous frame annotated image is predicted to be close to the type label of each pixel point in the previous frame annotated training image, the semantic segmentation model can label each pixel point in the previous frame image during the training process, and use the previous frame annotated image as a label to update the model parameters, and explicitly let the semantic segmentation model use the previous frame image and the previous frame annotated image to update the model parameters, thereby improving the prediction accuracy of the semantic segmentation model.

[0077] In some embodiments of the present application, based on Figure 2 The architecture of the semantic segmentation model of the example, the embodiment of the present application provides a flowchart of the prediction method of the semantic segmentation model, as shown in FIG. Figure 4 As shown, the method may include the following steps:

[0078] Step S300: input the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model.

[0079] Specifically, in this step, the trained semantic segmentation model can be used to perform semantic segmentation on the current frame image. The input layer of the semantic segmentation model can vectorize the current frame image, the previous frame image, and the previous frame annotated image, and output the vectorized current frame image, the previous frame image, and the previous frame annotated image to the current frame image annotation layer.

[0080] Step S301 : using the current frame image annotation layer of the semantic segmentation model, annotate the current frame image based on the previous frame image and the previous frame annotated image, so as to output the current frame annotated image.

[0081] Specifically, when the adjacent frames in the video to be annotated are not very different, since there are many common points between the current frame and the previous frame, the current frame image annotation layer in the semantic segmentation model can combine the previous frame and the previous annotated image to annotate the type of each pixel in the current frame, and finally output the current frame annotated image. Among them, each pixel in the current frame annotated image is annotated with a type.

[0082] In the above embodiment, the current frame image annotation layer of the semantic segmentation model can use the previous frame image and the previous annotated frame image as references when annotating the type of each pixel point in the current frame image. This makes it possible to improve the accuracy of annotation to a certain extent when annotating the types of each pixel point in the current frame image when there is little difference between adjacent frame images in the video to be annotated. Since there will be more similarities between the current frame image and the previous frame image, the accuracy of annotation can be improved to a certain extent.

[0083] Furthermore, before using the semantic segmentation model, a trained semantic segmentation model can be configured in the local host or on a remote server accessible to the local host. On this basis, when the trained semantic segmentation model is configured in the local host, the prediction task can be completed in the local host, and the processing is more efficient, but the local host configuration requirements are higher; and when the trained semantic segmentation model is configured on a remote server accessible to the local host, it is necessary to transmit the model input to the remote server configured with the semantic segmentation model through the network, and after the remote server completes the processing, the processed data is transmitted back to the local host through the network, where the delay in network transmission may affect the processing efficiency, but the configuration requirements for the local host are relatively low. Which configuration method to use shall be subject to the actual situation. Regardless of which configuration method is adopted, it will not affect the implementation of this application.

[0084] Next, combine Figure 5 As shown, another optional implementation method of semantic segmentation provided by an embodiment of the present application is introduced, and the method may include the following steps:

[0085] Step S400: Obtain the current frame image, the previous frame image and the previous annotated frame image of the video to be annotated.

[0086] Step S401: input the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain the current frame annotated image.

[0087] Among them, steps S400-S401 correspond to steps S100-S101 in the aforementioned embodiment, and please refer to the aforementioned introduction for details, which will not be repeated here.

[0088] Step S402: Determine whether each frame image in the video to be annotated has a corresponding annotated image.

[0089] Specifically, by judging whether each frame image in the video to be annotated has a corresponding annotated image, it can be determined whether the annotation has been completed. If some frame images in the video to be annotated do not have a corresponding annotated image, step S403 is executed; if each frame image in the video to be annotated has a corresponding annotated image, step S404 is executed.

[0090] Step S403: taking the current frame image as the previous frame image, the current frame annotated image as the previous frame annotated image, and taking the next frame image in the video to be annotated as the current frame image.

[0091] Specifically, the current frame annotated image obtained in the above steps can be used as the input of the semantic segmentation model in the next prediction cycle, so as to predict the type of each pixel in the next frame image. That is, the current frame image is used as the previous frame image, the current frame annotated image is used as the previous frame annotated image, and the next frame image in the video to be annotated is used as the current frame image, and the process returns to step S401, and the current frame image, the previous frame image, and the previous frame annotated image are input into the semantic segmentation model.

[0092] Step S404: End labeling.

[0093] Specifically, at this time, the annotated image corresponding to each frame image in the video to be annotated can be obtained.

[0094] In the above embodiment, after determining the current frame image in the video to be annotated and predicting the current frame annotated image through the semantic segmentation model, the annotation of other frame images after the current frame image in the video to be annotated can be automatically completed, thereby improving the annotation speed to a certain extent.

[0095] In some embodiments of the present application, reference Figure 6 As shown, another optional implementation method of semantic segmentation is introduced, which includes the following steps:

[0096] Step S500: Obtain the current frame image, the previous frame image and the previous annotated frame image of the video to be annotated.

[0097] Step S501: input the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain the current frame annotated image.

[0098] Among them, steps S500-S501 correspond to steps S100-S101 in the aforementioned embodiment, and please refer to the aforementioned introduction for details, which will not be repeated here.

[0099] Step S502: Record the number of annotated images in the video to be annotated.

[0100] Specifically, by inputting the current frame image, the previous frame image and the previous annotated image of the video to be annotated into the semantic segmentation model, the current frame annotated image can be obtained. After obtaining the current frame annotated image, since the current frame annotated image has been annotated by the semantic segmentation model, it can be used as an annotated image in the video to be annotated. Therefore, the number of annotated images in the recorded video to be annotated can be increased by one.

[0101] Step S503: determine whether the number is a multiple of a preset threshold.

[0102] Specifically, in the above steps, the number of annotated images in the video to be annotated can be obtained, and it is determined whether the number is a multiple of a preset threshold. When the number of annotated images in the video to be annotated is a multiple of the preset threshold, step S504 is executed.

[0103] Step S504: determine whether the user has any objection to the annotated image of the current frame.

[0104] Specifically, when the number of annotated images in the video to be annotated is a multiple of a preset threshold, verification information sent by the user can be received, and based on the verification information sent by the user, it can be determined whether the user has any objection to the annotated image of the current frame. The verification information sent by the user can be the judgment made by the user on the annotated image of the current frame after checking the annotated image of the current frame obtained in the current prediction cycle. When the user has an objection to the annotated image of the current frame based on the verification information, step S505 is executed; when the user has no objection to the annotated image of the current frame based on the verification information, the annotated image of the current frame output by the semantic segmentation model in the current prediction cycle is used as the final annotated image.

[0105] Step S505: Obtain the current frame annotated image modified by the user, and replace the current frame annotated image output by the semantic segmentation model in the current prediction cycle.

[0106] Specifically, since the user has objections to the current frame annotated image, the current frame annotated image modified by the user can be obtained and replaced with the current frame annotated image output by the semantic segmentation model in the current prediction cycle.

[0107] Step S506: Determine whether each frame image in the video to be annotated has a corresponding annotated image.

[0108] Specifically, by judging whether each frame image in the video to be annotated has a corresponding annotated image, it can be determined whether the annotation has been completed. If some frame images in the video to be annotated do not have a corresponding annotated image, step S507 is executed; if each frame image in the video to be annotated has a corresponding annotated image, step S508 is executed.

[0109] Step S507: taking the current frame image as the previous frame image, the current frame annotated image as the previous frame annotated image, and taking the next frame image in the to-be-annotated video as the current frame image.

[0110] Specifically, in the above steps, after the user modifies the current frame annotated image output by the semantic segmentation model, the current frame annotated image output by the semantic segmentation model in the current prediction cycle can be replaced. At this time, the current frame image can be used as the previous frame image, the replaced current frame annotated image can be used as the previous frame annotated image, and the next frame image in the video to be annotated can be used as the current frame image, and the process returns to step S501, where the current frame image, the previous frame image, and the previous frame annotated image are input into the semantic segmentation model.

[0111] For example, assuming the preset threshold is 5, when the number of annotated images in the video to be annotated reaches 5, 10, 15, 20, or other multiples of 5, the user can check the annotated image of the current frame in the current prediction period. If the user has any objection to the annotated image of the current frame, the user can modify the annotated image of the current frame and replace the previous annotated image of the current frame with the modified annotated image of the current frame.

[0112] Step S507: End labeling.

[0113] Specifically, at this time, the annotated image corresponding to each frame image in the video to be annotated can be obtained.

[0114] In the above embodiment, since machine labeling may have errors, after a certain number of cycles, the errors will continue to accumulate, and may eventually cause too many incorrectly labeled pixels in the current frame labeling image output by the semantic segmentation model and fail to meet the requirements. Therefore, after a certain number of cycles of labeling, manual intervention can be performed to judge the current frame labeling image, and the objectionable current frame labeling image can be modified to obtain the current frame labeling image, and the current frame labeling image output by the semantic segmentation model in the current prediction cycle can be replaced, and the replaced current frame labeling image can be used as an input to the semantic segmentation model in the next prediction cycle, thereby ensuring the accuracy of the labeling to a certain extent.

[0115] The following is a description of a semantic segmentation device provided in an embodiment of the present application. The semantic segmentation device described below and the semantic segmentation method described above can be referenced to each other.

[0116] Figure 7 A schematic diagram of a semantic segmentation device structure provided in an embodiment of the present application, see Figure 7 As shown, the semantic segmentation device may include:

[0117] The image acquisition unit 10 is used to acquire the current frame image, the previous frame image and the previous annotated image of the video to be annotated, wherein the previous annotated image includes the type of each pixel point;

[0118] The image annotation unit 20 is used to input the current frame image, the previous frame image and the previous frame annotation image into the semantic segmentation model to obtain the current frame annotation image;

[0119] Among them, the semantic segmentation model uses the current frame training image, the previous frame training image and the previous frame annotated training image as training samples, and is trained with the type of each pixel annotated in the current frame training image as the training label.

[0120] In the above embodiment, a semantic segmentation device is provided, which obtains the current frame image, the previous frame image and the previous frame annotated image containing the type of each pixel point of the video to be annotated through the image acquisition unit 10, and the image annotation unit 20 inputs the current frame image, the previous frame image and the previous frame annotated image into the semantic segmentation model to obtain the current frame annotated image, thereby realizing semantic segmentation of the image in the video to be annotated, wherein the semantic segmentation model uses the current frame training image, the previous frame training image and the previous frame training annotated image as training samples, and is trained with the type of each pixel point annotated in the current frame training image as training labels.

[0121] Furthermore, in the present application, when semantic segmentation is performed on the current frame image, the image annotation unit 20 also inputs the previous frame image and the previous annotated frame image as input together with the current frame image into the semantic segmentation model. When the difference between adjacent frame images in the video to be annotated is not large, the semantic segmentation model can combine the previous frame image and the previous annotated frame image to annotate the type of each pixel point in the current frame image, thereby improving the accuracy of the annotation to a certain extent.

[0122] Optionally, the semantic segmentation device may further include:

[0123] A model training unit, used for training a semantic segmentation model, wherein the semantic segmentation model includes: an input layer, a current frame image annotation layer, and a previous frame image annotation layer that are cascaded in sequence;

[0124] Among them, the training process of the semantic segmentation model includes:

[0125] Through the input layer, obtain the current frame training image, the previous frame training image and the previous frame annotated training image;

[0126] Through the current frame image annotation layer, based on the previous frame training image and the previous frame annotated training image, the current frame training image is annotated, and the current frame annotated image is predicted;

[0127] The previous frame of training image is annotated through the previous frame image annotation layer, and the previous frame of annotated image is predicted;

[0128] The parameters of the semantic segmentation model are updated with the predicted type of each pixel in the current frame annotated image being close to the type label of each pixel in the current frame training image as the training goal, and the predicted type of each pixel in the previous frame annotated image being close to the type label of each pixel in the previous frame annotated training image as the training goal.

[0129] Optionally, the semantic segmentation device may further include:

[0130] The cyclic annotation unit is used to take the current frame image as the previous frame image, the current frame annotated image as the previous frame annotated image, and the next frame image in the video to be annotated as the current frame image, and return to execute the step of inputting the current frame image, the previous frame image and the previous frame annotated image into the semantic segmentation model until each frame image in the video to be annotated corresponds to an annotated image.

[0131] Optionally, the process in which the image annotation unit 20 inputs the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain the current frame annotated image may include:

[0132] Inputting the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model;

[0133] The current frame image is annotated using the current frame image annotation layer of the semantic segmentation model based on the previous frame image and the previous frame annotated image to output the current frame annotated image.

[0134] Optionally, the semantic segmentation device may further include:

[0135] A quantity recording unit, used to record the number of labeled images in the video to be labeled;

[0136] A judgment unit is used to judge whether the number is a multiple of a preset threshold; if the number is a multiple of the preset threshold, receiving verification information sent by the user, and judging whether the user has any objection to the current frame annotated image based on the verification information; if the user has any objection to the current frame annotated image, obtaining the current frame annotated image modified by the user, and replacing the current frame annotated image output by the semantic segmentation model in the current prediction cycle.

[0137] The present application also provides a semantic segmentation device. Figure 8 The hardware structure diagram of the semantic segmentation device is shown in FIG. Figure 8 ,The hardware structure of the semantic segmentation device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0138] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0139] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0140] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0141] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to: implement each processing flow in the aforementioned semantic segmentation method.

[0142] An embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to: implement each processing flow in the aforementioned semantic segmentation method.

[0143] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0144] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined with each other, and the same or similar parts can be referenced to each other.

[0145] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantic segmentation method, It is characterized in that include: Obtaining a current frame image, a previous frame image, and a previous annotated image of the video to be annotated, wherein the previous annotated image contains the type of each pixel point; Input the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain a current frame annotated image; The semantic segmentation model is trained using the current frame training image, the previous frame training image and the previous frame annotated training image as training samples, and the type of each pixel annotated in the current frame training image as a training label; The semantic segmentation model includes: an input layer, a current frame image annotation layer and a previous frame image annotation layer which are cascaded in sequence; The training process of the semantic segmentation model includes: Through the input layer, obtain the current frame training image, the previous frame training image and the previous frame annotated training image; Through the current frame image annotation layer, based on the previous frame training image and the previous frame annotated training image, the current frame training image is annotated, and the current frame annotated image is predicted; The previous frame of training image is annotated through the previous frame image annotation layer, and the previous frame of annotated image is predicted; The parameters of the semantic segmentation model are updated with the predicted type of each pixel in the current frame annotated image being close to the type label of each pixel in the current frame training image as the training goal, and the predicted type of each pixel in the previous frame annotated image being close to the type label of each pixel in the previous frame annotated training image as the training goal.

2. The method according to claim 1, It is characterized in that The step of inputting the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain the current frame annotated image includes: Inputting the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model; The current frame image is annotated using the current frame image annotation layer of the semantic segmentation model based on the previous frame image and the previous frame annotated image to output the current frame annotated image.

3. The method according to claim 1, It is characterized in that Also includes: The current frame image is used as the previous frame image, the current frame annotated image is used as the previous frame annotated image, and the next frame image in the video to be annotated is used as the current frame image, and the step of inputting the current frame image, the previous frame image, and the previous frame annotated image into the semantic segmentation model is returned until each frame image in the video to be annotated corresponds to a labeled image.

4. The method according to claim 3, It is characterized in that Also includes: Record the number of labeled images in the video to be labeled; Determining whether the number is a multiple of a preset threshold; If the number is a multiple of the preset threshold, receiving verification information sent by the user, and judging whether the user has any objection to the annotated image of the current frame based on the verification information; If the user has any objection to the current frame annotated image, the current frame annotated image modified by the user is obtained and replaced with the current frame annotated image output by the semantic segmentation model in the current prediction cycle.

5. A semantic segmentation device, It is characterized in that include: An image acquisition unit, used to acquire a current frame image, a previous frame image and a previous annotated image of the video to be annotated, wherein the previous annotated image includes the type of each pixel point; An image annotation unit, used for inputting the current frame image, the previous frame image and the previous frame annotated image into a semantic segmentation model to obtain a current frame annotated image; The semantic segmentation model is trained using the current frame training image, the previous frame training image and the previous frame annotated training image as training samples, and the type of each pixel annotated in the current frame training image as a training label; A model training unit, used for training to obtain the semantic segmentation model, wherein the semantic segmentation model comprises: an input layer, a current frame image annotation layer, and a previous frame image annotation layer cascaded in sequence; The training process of the semantic segmentation model includes: Through the input layer, obtain the current frame training image, the previous frame training image and the previous frame annotated training image; Through the current frame image annotation layer, based on the previous frame training image and the previous frame annotated training image, the current frame training image is annotated, and the current frame annotated image is predicted; The previous frame of training image is annotated through the previous frame image annotation layer, and the previous frame of annotated image is predicted; The parameters of the semantic segmentation model are updated with the predicted type of each pixel in the current frame annotated image being close to the type label of each pixel in the current frame training image as the training goal, and the predicted type of each pixel in the previous frame annotated image being close to the type label of each pixel in the previous frame annotated training image as the training goal.

6. The device according to claim 5, It is characterized in that Also includes: The cyclic annotation unit is used to take the current frame image as the previous frame image, the current frame annotated image as the previous frame annotated image, and the next frame image in the video to be annotated as the current frame image, and return to execute the step of inputting the current frame image, the previous frame image, and the previous frame annotated image into the semantic segmentation model until each frame image in the video to be annotated corresponds to a labeled image.

7. A semantic segmentation device, It is characterized in that include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the semantic segmentation method as claimed in any one of claims 1 to 4.

8. A readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, each step of the semantic segmentation method according to any one of claims 1 to 4 is implemented.