Real-time semantic segmentation method and system based on spatial pyramid pooling

Abstract

The present invention relates to a real-time semantic segmentation method and system based on spatial pyramid pooling, the method comprising the following steps: step S1: obtaining a city streetscape data set Cityscapes and preprocessing it; step S2: performing offline data enhancement on the data set obtained in step S1 to increase the diversity of the data set; step S3: constructing a real-time semantic segmentation network LSPPNet based on spatial pyramid pooling; step S4: using the data set obtained in step S2 to train the LSPPNet neural network model; step S5: preprocessing the images collected in the autonomous driving scene, and then inputting them into the LSPPNet neural network model to obtain the image semantic segmentation result. The method and system are conducive to improving the accuracy and timeliness of image semantic segmentation.

Description

Technical Field

[0001] The invention belongs to the field of pattern recognition and computer vision, and in particular relates to a real-time semantic segmentation method and system based on spatial pyramid pooling. Background Art

[0002] In recent years, the development of computer hardware, especially the improvement of GPU computing power, has greatly promoted the progress of artificial intelligence and provided great opportunities for the development of computer vision. In computer vision tasks, image processing has benefited greatly, including target detection tasks, target recognition tasks, target tracking tasks, and image segmentation tasks. In these directions, the key role of image segmentation is to detect and analyze the given image data, identify the objects in the image, and then achieve a segmentation effect by dividing the target and background in the image. As a basic research direction in computer vision tasks, image segmentation technology has laid the foundation for more computer vision tasks. The premise for solving a large number of image processing tasks is to handle the results of image segmentation. In autonomous driving, the semantic segmentation model needs to be deployed to the edge device. At this time, the model is required to maintain high performance and perform reasoning at a faster speed. How to achieve a good compromise between the two indicators of speed and performance is a very challenging problem. Summary of the invention

[0003] The object of the present invention is to provide a real-time semantic segmentation method and system based on spatial pyramid pooling, which are conducive to improving the accuracy and timeliness of image semantic segmentation.

[0004] To achieve the above object, the technical solution adopted by the present invention is: a real-time semantic segmentation method based on spatial pyramid pooling, comprising the following steps:

[0005] Step S1: Obtain the city street view dataset Cityscapes and perform preprocessing;

[0006] Step S2: Perform offline data enhancement on the data set obtained in step S1 to increase the diversity of the data set;

[0007] Step S3: Construct a real-time semantic segmentation network LSPPNet based on spatial pyramid pooling;

[0008] Step S4: Use the data set obtained in step S2 to train the LSPPNet neural network model;

[0009] Step S5: Preprocess the images collected in the autonomous driving scene and then input them into the LSPPNet neural network model to obtain the image semantic segmentation results.

[0010] Furthermore, the step S1 specifically includes the following steps:

[0011] Step S11: Download the gtFine_trainvaltest.zip label data compression file and the leftImg8bit_trainvaltest.zip image data compression file from the open source dataset website cityscapes-dataset.com;

[0012] Step S12: decompress the two compressed files obtained in step S11, sort out and delete the label files that are not related to the semantic segmentation task.

[0013] Furthermore, the step S2 specifically includes the following steps:

[0014] Step S21: Flip all images in the data set obtained in step S1, and the corresponding label images are changed accordingly, and the flipped images are added to the new data set;

[0015] Step S22: Perform color dithering on all the images in the data set obtained in step S1, and the corresponding label images remain unchanged, and the color dithered images are added to the new data set;

[0016] Step S23: performing translation transformation on all images in the data set obtained in step S1, and the corresponding label images are changed accordingly, and the translated images are added to the new data set;

[0017] Step S24: performing contrast transformation on all images in the data set obtained in step S1, and the corresponding label images remain unchanged, and adding the contrast-transformed images to the new data set;

[0018] Step S25: Scale all images in the data set obtained in step S1, and the corresponding label images are changed accordingly, and the scale-transformed images are added to the new data set.

[0019] Furthermore, the step S3 specifically includes the following steps:

[0020] Step S31: ResNet-18 is used as the encoder of LSPPNet, where the bottleneck layer of ResNet-18 downsamples the input image by 4 times, and then ResNet-18 downsamples the image by 2 times in the next three stages except the first stage;

[0021] Step S32: In the first to third stages of ResNet-18, the representation of the downsampling stage is saved through jump connections, and a deep supervision module is introduced to supervise the representation of the fourth stage output of the encoder to reduce the loss of spatial information in the encoding stage;

[0022] Step S33: input the output result of the fourth stage of the encoder into the lightweight pyramid pooling module LSPPM to obtain a representation with rich multi-scale information;

[0023] Step S34: input the representation obtained in step S33 into the cascade upsampling path, and combine with the dense connection in step 32, use the fast upsampling module FUM to perform 2x upsampling on the representation for a total of 3 times, and finally obtain a representation that integrates semantic information and spatial information;

[0024] Step S35: The representation obtained in step S34 is upsampled 8 times through a bilinear interpolation algorithm, and a prediction result is output through a 1×1 convolution.

[0025] Furthermore, the step S4 specifically includes the following steps:

[0026] Step S41: Train the LSPPNet neural network model constructed in step S3, and set the initial parameters as follows:

[0027] Initial learning rate, i.e. -learning rate: 0.01;

[0028] Weight decay: 0.0005;

[0029] Momentum: 0.9

[0030] In the training phase, polynomial weight decay is used as the learning rate decay strategy, where the minimum learning rate is set to 0.0001, the decay factor is set to 0.9, and the batch size is determined according to the video memory size of the training server;

[0031] Step S42: The final loss function of the model is:

[0032] Loss final =Loss main +α·Loss aux

[0033] Among them, Loss final , Loss main , Loss aux They represent the final loss, main loss and auxiliary loss of the model respectively, where α is the weight of the auxiliary loss, which is set to 0.4, and the loss function uses cross information entropy;

[0034] Step S43: In the training phase, the stochastic gradient descent method is used as an optimizer to calculate the updated weight values ​​and bias values ​​of the convolutional neural network;

[0035] Step S44: performing random perspective transformation on some training samples, and making corresponding transformation on the label file, and then adding them to the training samples of the model to participate in the training;

[0036] Step S45: Gaussian blurring of random positions is performed on some training samples, and the corresponding label files remain unchanged, and then they are added to the training samples of the model for training;

[0037] Step S46: Stop training after 160,000 iterations and save the trained model.

[0038] Furthermore, the step S5 specifically includes the following steps:

[0039] Step S51: collecting image data as input through a vehicle-mounted camera in an autonomous driving scenario;

[0040] Step S52: resizing the input image to a size of 2048×1024;

[0041] Step S53: The image obtained in step S52 is subjected to the LSPPNet neural network model to obtain a prediction result graph;

[0042] Step S54: The prediction result image obtained in step S53 is scaled to the original input size using a bilinear interpolation algorithm to obtain a final result image.

[0043] The present invention also provides a real-time semantic segmentation system based on spatial pyramid pooling, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0044] Compared with the prior art, the present invention has the following beneficial effects: a real-time semantic segmentation method and system based on spatial pyramid pooling are provided. Compared with the traditional method, the present invention can effectively perform semantic segmentation for autonomous driving scenarios, improve the effect of image segmentation, and has a relatively good speed in processing high-resolution image data, and is suitable for deployment in terminal devices with limited performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a method implementation of an embodiment of the present invention;

[0046] Figure 2 It is a network structure diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0048] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0050] like Figure 1 As shown, this embodiment provides a real-time semantic segmentation method based on spatial pyramid pooling, comprising the following steps:

[0051] Step S1: Obtain the city street view dataset Cityscapes and perform preprocessing.

[0052] Step S2: Perform offline data enhancement on the dataset obtained in step S1 to increase the diversity of the dataset.

[0053] Step S3: Construct a real-time semantic segmentation network LSPPNet based on spatial pyramid pooling.

[0054] Step S4: Use the data set obtained in step S2 to train the LSPPNet neural network model.

[0055] Step S5: Preprocess the images collected in the autonomous driving scene and then input them into the LSPPNet neural network model to obtain the image semantic segmentation results.

[0056] In this embodiment, step S1 specifically includes the following steps:

[0057] Step S11: Download the gtFine_trainvaltest.zip label data compression file and the leftImg8bit_trainvaltest.zip image data compression file from the open source dataset website cityscapes-dataset.com;

[0058] Step S12: decompress the two compressed files obtained in step S11, sort out and delete the label files that are not related to the semantic segmentation task.

[0059] In this embodiment, step S2 specifically includes the following steps:

[0060] Step S21: Flip all images in the data set obtained in step S1, and the corresponding label images are changed accordingly, and the flipped images are added to the new data set;

[0061] Step S22: Perform color dithering on all the images in the data set obtained in step S1, and the corresponding label images remain unchanged, and the color dithered images are added to the new data set;

[0062] Step S23: performing translation transformation on all images in the data set obtained in step S1, and the corresponding label images are changed accordingly, and the translated images are added to the new data set;

[0063] Step S24: performing contrast transformation on all images in the data set obtained in step S1, and the corresponding label images remain unchanged, and adding the contrast-transformed images to the new data set;

[0064] Step S25: Scale all images in the data set obtained in step S1, and the corresponding label images are changed accordingly, and the scale-transformed images are added to the new data set.

[0065] In this embodiment, step S3 specifically includes the following steps:

[0066] Step S31: ResNet-18 is used as the encoder of LSPPNet, where the bottleneck layer of ResNet-18 downsamples the input image by 4 times, and then ResNet-18 downsamples the image by 2 times in the next three stages except the first stage;

[0067] Step S32: In the first to third stages of ResNet-18, the representation of the downsampling stage is saved through jump connections, and a deep supervision module is introduced to supervise the representation of the fourth stage output of the encoder to reduce the loss of spatial information in the encoding stage;

[0068] Step S33: input the output result of the fourth stage of the encoder into the lightweight pyramid pooling module LSPPM to obtain a representation with rich multi-scale information;

[0069] Step S34: input the representation obtained in step S33 into the cascade upsampling path, and combine with the dense connection in step 32, use the fast upsampling module FUM to perform 2x upsampling on the representation for a total of 3 times, and finally obtain a representation that integrates semantic information and spatial information;

[0070] Step S35: The representation obtained in step S34 is upsampled 8 times through a bilinear interpolation algorithm, and a prediction result is output through a 1×1 convolution.

[0071] The network structure constructed in this embodiment is as follows Figure 2 shown.

[0072] In this embodiment, step S4 specifically includes the following steps:

[0073] Step S41: Train the LSPPNet neural network model constructed in step S3, and set the initial parameters as follows:

[0074] Initial learning rate, i.e. -learning rate: 0.01;

[0075] Weight decay: 0.0005;

[0076] Momentum: 0.9

[0077] In the training phase, polynomial weight decay is used as the learning rate decay strategy, where the minimum learning rate is set to 0.0001, the decay factor is set to 0.9, and the batch size is determined according to the video memory size of the training server;

[0078] Step S42: The final loss function of the model is:

[0079] Loss final =Loss main +Loss aux

[0080] Among them, Loss final , Loss main , Loss aux They represent the final loss, main loss and auxiliary loss of the model respectively, where α is the weight of the auxiliary loss, which is set to 0.4, and the loss function uses cross information entropy;

[0081] Step S43: In the training phase, the stochastic gradient descent method is used as an optimizer to calculate the updated weight values ​​and bias values ​​of the convolutional neural network;

[0082] Step S44: performing random perspective transformation on some training samples, and making corresponding transformation on the label file, and then adding them to the training samples of the model to participate in the training;

[0083] Step S45: Gaussian blurring of random positions is performed on some training samples, and the corresponding label files remain unchanged, and then they are added to the training samples of the model for training;

[0084] Step S46: Stop training after 160,000 iterations and save the trained model.

[0085] In this embodiment, step S5 specifically includes the following steps:

[0086] Step S51: collecting image data as input through a vehicle-mounted camera in an autonomous driving scenario;

[0087] Step S52: resizing the input image to a size of 2048×1024;

[0088] Step S53: The image obtained in step S52 is subjected to the LSPPNet neural network model to obtain a prediction result graph;

[0089] Step S54: The prediction result image obtained in step S53 is scaled to the original input size using a bilinear interpolation algorithm to obtain a final result image.

[0090] This embodiment also provides a real-time semantic segmentation system based on spatial pyramid pooling, including a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the above method steps can be implemented.

[0091] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0092] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0093] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0095] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A real-time semantic segmentation method based on spatial pyramid pooling, characterized in that: The following steps are involved: Step S1: Obtain the city street view dataset Cityscapes and perform preprocessing; Step S2: Perform offline data enhancement on the data set obtained in step S1 to increase the diversity of the data set; Step S3: Construct a real-time semantic segmentation network LSPPNet based on spatial pyramid pooling; Step S4: Use the data set obtained in step S2 to train the LSPPNet neural network model; Step S5: pre-process the images collected in the autonomous driving scene, and then input them into the LSPPNet neural network model to obtain the image semantic segmentation results; The step S2 specifically includes the following steps: Step S21: Flip all images in the data set obtained in step S1, and the corresponding label images are changed accordingly, and the flipped images are added to the new data set; Step S22: Perform color dithering on all the images in the data set obtained in step S1, and the corresponding label images remain unchanged, and the color dithered images are added to the new data set; Step S23: performing translation transformation on all images in the data set obtained in step S1, and the corresponding label images are changed accordingly, and the translated images are added to the new data set; Step S24: performing contrast transformation on all images in the data set obtained in step S1, and the corresponding label images remain unchanged, and adding the contrast-transformed images to the new data set; Step S25: scale all the images in the data set obtained in step S1, and the corresponding label images are changed accordingly, and the scale-transformed images are added to the new data set; The step S3 specifically comprises the following steps: Step S31: ResNet-18 is used as the encoder of LSPPNet, where the bottleneck layer of ResNet-18 downsamples the input image by 4 times, and then ResNet-18 downsamples the image by 2 times in the next three stages except the first stage; Step S32: In the first to third stages of ResNet-18, the representation of the downsampling stage is saved through jump connections, and a deep supervision module is introduced to supervise the representation of the fourth stage output of the encoder to reduce the loss of spatial information in the encoding stage; Step S33: input the output result of the fourth stage of the encoder into the lightweight pyramid pooling module LSPPM to obtain a representation with rich multi-scale information; Step S34: input the representation obtained in step S33 into the cascade upsampling path, and combine with the dense connection in step 32, use the fast upsampling module FUM to perform 2x upsampling on the representation for a total of 3 times, and finally obtain a representation that integrates semantic information and spatial information; Step S35: The representation obtained in step S34 is upsampled 8 times through a bilinear interpolation algorithm, and a prediction result is output through a 1×1 convolution.

2. The real-time semantic segmentation method based on spatial pyramid pooling according to claim 1, characterized in that: The step S1 specifically includes the following steps: Step S11: Download the gtFine_trainvaltest.zip label data compression file and the leftImg8bit_trainvaltest.zip image data compression file from the open source dataset website cityscapes-dataset.com; Step S12: decompress the two compressed files obtained in step S11, sort out and delete the label files that are not related to the semantic segmentation task.

3. The real-time semantic segmentation method based on spatial pyramid pooling according to claim 1, characterized in that: The step S4 specifically comprises the following steps: Step S41: Train the LSPPNet neural network model constructed in step S3, and set the initial parameters as follows: Initial learning rate, i.e. -learning rate: 0.01; Weight decay: 0.0005; Momentum: 0.9 In the training phase, polynomial weight decay is used as the learning rate decay strategy, where the minimum learning rate is set to 0.0001, the decay factor is set to 0.9, and the batch size is determined according to the video memory size of the training server; Step S42: The final loss function of the model is: Loss final =Loss main +α·Loss aux Among them, Loss final , Loss main , Loss aux They represent the final loss, main loss and auxiliary loss of the model respectively, where α is the weight of the auxiliary loss, which is set to 0.4, and the loss function uses cross information entropy; Step S43: In the training phase, the stochastic gradient descent method is used as an optimizer to calculate the updated weight values ​​and bias values ​​of the convolutional neural network; Step S44: performing random perspective transformation on some training samples, and making corresponding transformation on the label file, and then adding them to the training samples of the model to participate in the training; Step S45: Gaussian blurring of random positions is performed on some training samples, and the corresponding label files remain unchanged, and then they are added to the training samples of the model for training; Step S46: Stop training after 160,000 iterations and save the trained model.

4. The real-time semantic segmentation method based on spatial pyramid pooling according to claim 1, characterized in that: The step S5 specifically comprises the following steps: Step S51: collecting image data as input through a vehicle-mounted camera in an autonomous driving scenario; Step S52: resizing the input image to a size of 2048×1024; Step S53: The image obtained in step S52 is subjected to the LSPPNet neural network model to obtain a prediction result graph; Step S54: The prediction result image obtained in step S53 is scaled to the original input size using a bilinear interpolation algorithm to obtain a final result image.

5. A real-time semantic segmentation system based on spatial pyramid pooling, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as described in any one of claims 1 to 4 can be implemented.

Images