A semantic segmentation method, device and equipment of a road scene image and a medium

Abstract

The application provides a semantic segmentation method, device and equipment of a road scene image and a medium. The method comprises receiving an image data set of a road scene, the image data set comprising image training data and image test data; pre-processing the image training data to generate an image sample data set; establishing an initial segmentation model based on a feature pyramid network; inputting the image sample data set into the initial segmentation model, training the initial segmentation model to generate a target segmentation model; and inputting the image test data into the target segmentation model for segmentation prediction processing to generate target image data. The application can realize semantic segmentation processing of a road scene image, and has high segmentation accuracy for small categories and small areas in a road scene.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a semantic segmentation method, apparatus, device, and medium for road scene images. Background Technology

[0002] Semantic segmentation of images has wide applications in fields such as autonomous driving and medical imaging. Semantic segmentation associates each pixel in an image with a category and assigns a category label; each pixel has exactly one category label. It divides and merges pixels with similar features into several non-overlapping regions based on deep and shallow features of the image.

[0003] In road scenes, image semantic segmentation divides road elements such as drivable areas, pedestrians, vehicles, and traffic signs into regions to support vehicles in acquiring environmental perception information. Some road scene categories occupy small areas in the image, resulting in lower model attention and poor segmentation performance for these categories.

[0004] Therefore, improving the accuracy of semantic segmentation models in small region segmentation, alleviating the imbalance in data sample distribution, and ensuring the accurate output of the model are urgent problems to be solved. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides a semantic segmentation method, apparatus, device and medium for road scene images to solve the above-mentioned technical problems.

[0006] This invention provides a semantic segmentation method for road scene images, including...

[0007] Receive an image dataset of a road scene, the image dataset including image training data and image test data;

[0008] The image training data is preprocessed to generate an image sample dataset;

[0009] An initial segmentation model is established based on the feature pyramid network.

[0010] The image sample dataset is input into the initial segmentation model to train the initial segmentation model, thereby generating the target segmentation model; and

[0011] The image test data is input into the target segmentation model for segmentation prediction processing to generate target image data.

[0012] In one embodiment of the present invention, the step of inputting the image sample dataset into the initial segmentation model and training the initial segmentation model includes:

[0013] The image samples in the image sample dataset are subjected to 2x, 4x, ..., 2x magnification. N Subsampling was performed to generate intermediate data C2, C4, ...

[0014] For the intermediate data Feature extraction is performed through convolution, upsampling is applied using bilinear interpolation, and then compared with the intermediate data. Add to generate a feature map

[0015] Following the steps of feature extraction via convolution, upsampling via bilinear interpolation, and addition to the intermediate data, the feature map is iteratively processed until feature map O2 is obtained; and

[0016] The feature map O2 is subjected to bilinear interpolation to generate the feature map of the image sample.

[0017] In one embodiment of the present invention, the step of inputting the image sample dataset into the initial segmentation model for training to generate a target segmentation model includes:

[0018] The initial segmentation model is trained with the following parameters: initial learning rate, weight decay parameter, number of iterations, initial warm-up learning rate, number of warm-up steps, exponential decay strategy for the learning rate, and model loss function; and

[0019] The image sample dataset is input into the initial segmentation model after configuration for training to generate the target segmentation model.

[0020] In one embodiment of the present invention, the model loss function includes class weights and class region weights, and the model loss function satisfies the following formula:

[0021] loss=weight-category×weight_region×L softmax_cross_entropy

[0022] Where loss represents the model loss function, weight_category represents the class weights, weight_region represents the class region weights, and L softmax_cross_entropy This represents the loss calculated from the output of the initial segmentation model.

[0023] In one embodiment of the present invention, the category weight represents the ratio of the number of images in each category to the total number of images in the image sample dataset, and the category weight satisfies the following formula:

[0024]

[0025] Where img_num represents the total number of images in the image sample dataset, and A(c) represents the number of images in each category of the image sample dataset.

[0026] In one embodiment of the present invention, the step of preprocessing the image training data to generate an image sample dataset includes:

[0027] Image samples are extracted from the image training data, the image samples representing road scene images; and

[0028] The image samples are semantically annotated to generate the image sample dataset.

[0029] In one embodiment of the present invention, the step of inputting the image test data into the target segmentation model for segmentation prediction processing to generate target image data includes:

[0030] The image test data is input into the target segmentation model for prediction calculation to generate image prediction data; and

[0031] The image prediction data and the image test data are fused together to generate the target image data.

[0032] The present invention also provides a semantic segmentation apparatus for road scene images, comprising:

[0033] The data receiving module is used to receive an image dataset of a road scene, the image dataset including image training data and image test data;

[0034] The data preprocessing module is used to preprocess the image training data to generate an image sample dataset;

[0035] The model building module is used to build an initial segmentation model based on the feature pyramid network;

[0036] A model training module is used to input the image sample dataset into the initial segmentation model, train the initial segmentation model, and generate a target segmentation model; and

[0037] The model testing module is used to input the image test data into the target segmentation model for segmentation prediction processing to generate target image data.

[0038] The present invention also provides an electronic device, comprising:

[0039] One or more processors;

[0040] A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to implement the semantic segmentation method for road scene images as described above.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer processor, causes the computer to perform the semantic segmentation method for road scene images described in any one of the above-described methods.

[0042] In summary, the present invention provides a semantic segmentation method, apparatus, device, and medium for road scene images, which can be used for semantic segmentation processing of road scene images, and the present invention has high segmentation accuracy for small categories and small regions in road scenes.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0045] Figure 1 This is the application environment of the semantic segmentation method for road scene images provided by the present invention;

[0046] Figure 2 This is a flowchart illustrating a semantic segmentation method for road scene images provided by the present invention;

[0047] Figure 3 yes Figure 2 A flowchart illustrating a specific implementation of step S202;

[0048] Figure 4 yes Figure 2 A flowchart illustrating a specific implementation of step S204;

[0049] Figure 5 yes Figure 2 A flowchart illustrating another specific implementation of step S204;

[0050] Figure 6 yes Figure 2 A schematic diagram of a specific implementation method for step S205;

[0051] Figure 7This is an overall flowchart of a semantic segmentation method for road scene images provided by the present invention;

[0052] Figure 8 This is a structural block diagram of a semantic segmentation device for road scene images provided by the present invention;

[0053] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention. Detailed Implementation

[0054] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0055] The illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0056] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0057] Please see Figure 1 As shown, Figure 1This invention provides an application environment for a semantic segmentation method for road scene images. Client 101 can communicate with server 102 via a network. First, server 102 receives an image dataset of the road scene acquired by an image acquisition device and annotates the acquired images. Then, it processes the image training data in the image dataset to generate an image sample dataset. Next, the image sample dataset is input into the initial segmentation model for training to generate a target segmentation model. The target segmentation model can be deployed on client 101. Image test data is input into client 101 for segmentation prediction processing to generate target image data. The invention is described in detail below through specific embodiments. Client 101 can be a smart car, and server 102 can be a physical server. This invention proposes a semantic segmentation method, apparatus, device, and medium for road scene images. Based on a multi-scale feature fusion network and trained with weight loss, it can improve the accuracy of the segmentation model in small region segmentation, alleviate the problem of imbalanced sample distribution, and ensure accurate model output. These embodiments will be described in detail below.

[0058] Please see Figure 2 As shown, Figure 2 This is a flowchart illustrating a semantic segmentation method for road scene images provided by the present invention, which may include the following steps:

[0059] Step S201: Receive the image dataset of the road scene. The image dataset includes image training data and image test data.

[0060] In one embodiment of the present invention, when step S201 is executed, an image dataset of a road scene is received. The image dataset includes image training data and image test data. Specifically, the image dataset of the road scene may include image training data and image test data. Each image of the road scene may contain road scene elements such as pedestrians, vehicles, and / or traffic signs. The image training data can be obtained by recording road scene videos to acquire a large amount of sample data. After preprocessing, the image training data can be used to train an initial model. The image test data can be collected by the front-facing camera of a car and can be used to test the trained model.

[0061] Step S202: Preprocess the image training data to generate an image sample dataset.

[0062] In one embodiment of the present invention, when step S202 is executed, the image training data is preprocessed to generate an image sample dataset. Specifically, the preprocessing includes image extraction and image annotation. The image training data obtained in step S201 is video format data, which first needs to be converted to generate image samples. In this embodiment, the format conversion adopts a frame extraction method at fixed intervals to extract frames from the recorded video and then obtain image samples. Then, the extracted image samples are labeled according to a preset segmentation and annotation document, and the labeled image samples are then processed by scaling, cropping, and category merging to generate an image sample dataset.

[0063] Step S203: Establish an initial segmentation model based on the feature pyramid network.

[0064] In one embodiment of the present invention, when step S203 is executed, an initial segmentation model is established based on the feature pyramid network. Specifically, the feature pyramid network, or FPN, can ensure that features at different scales contain rich semantic information without excessive computational cost. The feature pyramid network adopts a top-down and lateral connection approach, fusing high-resolution, low-semantic low-level features with low-resolution, high-semantic high-level features, while also fusing the original features. This results in feature maps at different scales containing rich semantic information; that is, top-level features are fused with low-level features through upsampling, and each layer performs independent prediction. Establishing an initial segmentation model based on the feature pyramid network can solve the multi-scale problem in image semantic segmentation. By changing the network connections, predictions can be performed independently at different feature layers without significantly increasing the computational cost of the original model, greatly improving the performance of the segmentation model for small object detection.

[0065] Step S204: Input the image sample dataset into the initial segmentation model and train the initial segmentation model to generate the target segmentation model.

[0066] In one embodiment of the present invention, when step S204 is executed, the image sample dataset is input into the initial segmentation model to train the initial segmentation model and generate the target segmentation model. Specifically, to train the initial segmentation model constructed in step S203, initial parameters must first be set. The initial parameters may include the initial learning rate lr. base Weight decay parameter w_dacay, iteration count max_iter, initial warmup learning rate warmup_lr baseThe initial segmentation model is configured with the following parameters: warmup step, learning rate (lr) with exponential decay, and model loss function. The model loss function can include class weights and class region weights. Then, the Adam optimization algorithm is used as the optimizer to update the weight values. Adam is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process; it iteratively updates the neural network weights based on the training data. Finally, the image sample dataset is input into the initial segmentation model for training. The trained initial segmentation model is then saved to obtain the target segmentation model.

[0067] Step S205: Input the image test data into the target segmentation model for segmentation prediction processing to generate target image data.

[0068] In one embodiment of the present invention, when step S205 is executed, the image test data is input into the target segmentation model for segmentation prediction processing to generate target image data. Specifically, the image test data collected by the vehicle-mounted front camera is first standardized. In this embodiment, the standardization processing can be performed by resizing and cropping the input image test data to adjust it to a standard size. Then, the standardized image test data is input into the target segmentation model generated in step S204 for segmentation prediction calculation to generate image prediction data. Finally, the image prediction data and the image test data are fused to generate the target image data.

[0069] Please see Figure 3 As shown, Figure 3 yes Figure 2 A schematic flowchart illustrating a specific implementation of step S202. Step S202 may include the following steps:

[0070] Step S301: Extract image samples from the image training data.

[0071] In one embodiment of the present invention, when step S301 is executed, image samples are extracted from the image training data. Specifically, the image training data is video format data, which first needs to be converted to generate image samples. In this embodiment, the format conversion adopts a frame-sampling method at fixed intervals to extract frames from the recorded video, thereby obtaining image samples.

[0072] Step S302: Perform semantic annotation processing on the image samples to generate an image sample dataset.

[0073] In one embodiment of the present invention, when step S302 is executed, semantic annotation processing is performed on the image samples to generate an image sample dataset. Specifically, according to a preset segmentation annotation document, data annotation processing is performed on the extracted images to generate an image sample dataset. In this embodiment, the segmentation annotation document defines the segmentation categories and segmentation methods of the images. The annotated image samples also need to undergo format standardization processing such as scaling, cropping, and category merging.

[0074] Please see Figure 4 As shown, Figure 4 yes Figure 2 A schematic flowchart illustrating a specific implementation of step S204. Step S204 may include the following steps:

[0075] Step S401: Set the training parameters for the initial segmentation model. The training parameters include the initial learning rate, weight decay parameter, number of iterations, initial warm-up learning rate, number of warm-up steps, exponential decay strategy of learning rate, and model loss function.

[0076] In one embodiment of the present invention, when step S401 is executed, the training parameters of the initial segmentation model are set. Specifically, to train the initial segmentation model constructed in step S203, the training parameters must first be set. The training parameters may include the initial learning rate, weight decay parameters, number of iterations, initial warm-up learning rate, number of warm-up steps, exponential decay strategy of the learning rate, and model loss function.

[0077] In this embodiment, the initial learning rate lr base The weight decay parameter w_dacay can be set to 0.005, the number of iterations max_iter can be set to 100000, and the initial warm-up learning rate warmup_lr can be set to 0.005. base It can be set to 0, and the warmup step can be set to 1500.

[0078] Learning rate decay strategies can be used to optimize training effectiveness and training time. Common learning rate decay strategies include piecewise constant decay, exponential decay, and polynomial decay. In this embodiment, an exponential decay strategy is used, which satisfies the following formula:

[0079]

[0080] Where lr represents the new learning rate, lr base `iter` represents the initial learning rate, `max_iter` represents the current iteration number, `max_iter` represents the maximum number of iterations, and `power` represents the rate at which the learning rate decreases.

[0081] The model loss function includes a category weight function and a category region weight function. First, the ratio of the number of images in each category to the total number of images can be calculated offline from the image sample dataset generated in step S202. This ratio is used as the category weight in the loss calculation. The category weights can satisfy the following formula:

[0082]

[0083] Where weight_category represents the category weight function, img_num represents the total number of images, and A(c) represents the number of images in each category.

[0084] Then, the ratio of the number of pixels in the corresponding category to the total number of pixels is calculated online, and this ratio is used as the category region weight in the loss calculation. The category region weight can satisfy the following formula:

[0085]

[0086] Where pixel_num is the total number of pixels, and P(c) is the number of pixels of the corresponding category in each image.

[0087] In this embodiment, the model loss function can satisfy the following formula:

[0088] loss=weight-category×weight_region×L softmax_cross_entropy

[0089] Where loss represents the model loss function, weight_category represents the class weights, weight_region represents the class region weights, and L softmax_cross_entropy This represents the loss obtained by applying cross-entropy to the model output after applying softmax. Softmax is a normalized exponential function that can represent the results of multi-class classification in the form of probabilities.

[0090] Step S402: Input the image sample dataset into the initial segmentation model after setting and train it to generate the target segmentation model.

[0091] In one embodiment of the present invention, when step S402 is executed, the initial segmentation model after inputting the image sample dataset is trained to generate the target segmentation model. Specifically, the initial segmentation model after inputting the image sample dataset is trained. In this embodiment, the Adam optimization algorithm is used as the optimizer to update the weight values. After inputting the image sample dataset into the initial segmentation model for training and saving the trained initial segmentation model, the target segmentation model is obtained.

[0092] Please see Figure 5 As shown, Figure 5 yes Figure 2 A schematic diagram of another specific embodiment of step S204 is shown. Step S204 may include the following steps:

[0093] Step S501: Perform image sample processing on the image sample dataset at multiples of 2x, 4x, ..., 2x. N Subsampling was performed to generate intermediate data C2, C4, ... .

[0094] In one embodiment of the present invention, when step S501 is executed, that is, the image samples in the image sample dataset are multiplied by 2, 4, ..., 2... N Subsampling was performed to generate intermediate data C2, C4, ... Specifically, based on the feature pyramid network, multiple downsampling network blocks are constructed. These downsampling network blocks are used to perform bottom-up feature convolution. Feature convolution is the forward propagation process of a neural network, representing the process of extracting features from an image input into the backbone convolutional network. Taking Residual Neural Network (ResNet) as an example, multiple downsampling network blocks (conv2, conv4, conv8, conv16, conv32, and conv64) perform downsampling processing on the feature map by 2x, 4x, 8x, 16x, 32x, and 64x, respectively. In this embodiment, the output of the above downsampling network blocks can be defined as {C2, C4, C8, C...} 16 C 32 C 64 These outputs have strides of {2, 4, 8, 16, 32, 64} relative to the input image, so they are {1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64} times the original image, i.e., reduced by a factor of 2. The outputs of the downsampling network blocks described above can be saved through skip connections.

[0095] Step S502: Process intermediate data Feature extraction is performed through convolution, upsampling is applied using bilinear interpolation, and then compared with intermediate data. Add to generate a feature map .

[0096] In one embodiment of the present invention, when step S502 is executed, that is, when intermediate data is processed... Feature extraction is performed through convolution, upsampling is applied using bilinear interpolation, and then compared with intermediate data. Add to generate a feature map Specifically, the feature pyramid network performs high-dimensional feature fusion on adjacent feature maps from top to bottom. The feature pyramid network then integrates the feature maps obtained from the higher levels. Upsampling is performed and then propagated downwards because high-level features contain rich semantic information. Top-down propagation ensures this semantic information reaches lower-level features, allowing them to also contain rich semantic information. For intermediate data... After performing convolution, feature extraction, and bilinear interpolation to complete upsampling, it is compared with intermediate data. Add to generate a feature map This invention employs bilinear interpolation to perform upsampling, thereby doubling the size of the feature map, i.e., magnifying it to the size of the feature map from the previous stage (step). Figure 1 The sample size is determined by using bilinear interpolation, which maximizes the preservation of semantic information in the feature map during upsampling, thus benefiting image classification. This interpolation is then fused with the corresponding feature maps, which possess rich spatial information and high resolution, facilitating localization, during low-to-high-resolution feature convolution. This results in feature maps that offer both good spatial information and strong semantic information. .

[0097] Step S503: Following the steps of extracting features by convolution, performing upsampling by bilinear interpolation, and adding the data to the intermediate data, iteratively process the feature map until feature map O2 is obtained.

[0098] In one embodiment of the present invention, when step S503 is executed, i.e., after extracting features by convolution, performing upsampling by bilinear interpolation, and adding the results to the intermediate data, the feature map is iteratively processed until feature map O2 is obtained. Specifically, taking N as a value of 6, the initial segmentation model downsamples the output C by 64 times. 64 Convolution is performed to further extract features, followed by bilinear interpolation for upsampling, and then the result is compared with the 32x downsampled output C. 32 Add them together to get the output O. 32 The output will be O. 32 Convolution is performed to further extract features, followed by bilinear interpolation for upsampling, and then the result is compared with the 16x downsampled output C. 16 Add them together to get the output O. 16 The output will be O. 16 Convolution is performed to further extract features, followed by bilinear interpolation for upsampling. This result is then added to the output C8, which has been downsampled by 8 times, to obtain output O8. Convolution is then performed on output O8 to further extract features, followed by bilinear interpolation for upsampling. This result is then added to the output C4, which has been downsampled by 4 times, to obtain output O4. Convolution is then performed on output O4 to further extract features, followed by bilinear interpolation for upsampling. This result is then added to the output C2, which has been downsampled by 2 times, to obtain output O2.

[0099] Step S504: Perform bilinear interpolation on feature map O2 to generate feature maps of image samples.

[0100] In one embodiment of the present invention, when step S404 is executed, bilinear interpolation is performed on feature map O2 to generate feature map of image sample. Specifically, bilinear interpolation is performed on O2 to output feature map of image sample, and the output feature map has the same size as the original image sample.

[0101] Please see Figure 6 As shown, Figure 6 yes Figure 2 A schematic flowchart illustrating a specific implementation of step S205. Step S205 may include the following steps:

[0102] Step S601: Input the image test data into the target segmentation model for prediction calculation to generate image prediction data.

[0103] In one embodiment of the present invention, when step S601 is executed, the image test data is input into the target segmentation model for prediction calculation to generate image prediction data. Specifically, the image test data collected by the vehicle-mounted camera is first standardized. In this embodiment, the standardization process can be performed by resizing and cropping the input image data to adjust it to a standard size. Then, the standardized image test data is input into the target segmentation model generated in step S204 for prediction calculation to generate image prediction data.

[0104] Step S602: The image prediction data and the image test data are fused to generate the target image data.

[0105] In one embodiment of the present invention, when step S602 is executed, the image prediction data and the image test data are fused to generate target image data. Specifically, since the image prediction data generated by the target segmentation model is the original image size, the target image prediction data and the target image data can be fused point-to-point to generate image segmentation data.

[0106] Please see Figure 7The diagram shows the overall flowchart of a semantic segmentation method for road scene images provided by the present invention. In one embodiment of the present invention, step S701 receives image training data for model training and image test data for model testing; step S702 preprocesses the image training data to form an image sample dataset; step S703 establishes an initial segmentation model using a feature pyramid network; step S704 inputs the image sample dataset into the initial segmentation model to train the initial segmentation model to generate a target segmentation model. During the training process, step S704 uses the Backpropagation algorithm to update the training parameters; step S705 inputs the processed image test data into the target segmentation model for segmentation prediction processing to generate a rendered image of the target image data.

[0107] Please see Figure 8 As shown, Figure 8 This is a structural block diagram of a semantic segmentation device for road scene images provided by the present invention. In some embodiments, the semantic segmentation device for road scene images includes a data receiving module 801, a data preprocessing module 802, a model building module 803, a model training module 804, and a model testing module 805. The functional modules are described in detail below.

[0108] The data receiving module 801 is used to receive an image dataset of a road scene, which includes image training data and image test data.

[0109] In one embodiment of the present invention, the data receiving module 801 can be used to receive an image dataset of a road scene, the image dataset including image training data and image test data. Specifically, the image dataset of the road scene may include image training data and image test data, and each image of the road scene may contain road scene elements such as pedestrians, vehicles, and / or traffic signs. The image training data can be obtained by recording road scene videos to acquire a large amount of sample data. After preprocessing, the image training data can be used to train an initial model. The image test data can be collected by a car's front-facing camera and can be used to test the trained model.

[0110] The data preprocessing module 802 is used to preprocess the image training data to generate an image sample dataset.

[0111] In one embodiment of the present invention, the data preprocessing module 802 can be used to preprocess image training data to generate an image sample dataset. Specifically, the preprocessing includes image extraction and image annotation. The image training data received by the data receiving module 801 is video format data, which first needs to be converted to generate image samples. In this embodiment, the format conversion adopts a frame extraction method at fixed intervals to extract frames from the recorded video, thereby obtaining image samples. Then, the extracted image samples are labeled according to a preset segmentation and annotation document, and the labeled image samples are then subjected to scaling, cropping, category merging, and other processing to generate an image sample dataset.

[0112] In one specific embodiment, the data preprocessing module 802 may also be specifically used for:

[0113] Image samples are extracted from the image training data; these image samples represent road scene images.

[0114] Semantic annotation is performed on image samples to generate an image sample dataset.

[0115] Model building module 803 is used to build an initial segmentation model based on the feature pyramid network.

[0116] In one embodiment of the present invention, the model building module 803 can be used to establish an initial segmentation model based on a feature pyramid network. Specifically, a feature pyramid network (FPN) can ensure that features at different scales contain rich semantic information without excessive computational cost. The feature pyramid network adopts a top-down and lateral connection approach, fusing high-resolution, low-semantic low-level features with low-resolution, high-semantic high-level features, while also fusing the original features. This results in feature maps at different scales containing rich semantic information; that is, top-level features are fused with low-level features through upsampling, and each layer makes independent predictions. Establishing an initial segmentation model based on a feature pyramid network can solve the multi-scale problem in image semantic segmentation. By changing the network connections, predictions can be made independently at different feature layers without significantly increasing the computational cost of the original model, greatly improving the performance of the segmentation model for small object detection.

[0117] The model training module 804 is used to input the image sample dataset into the initial segmentation model, train the initial segmentation model, and generate the target segmentation model.

[0118] In one embodiment of the present invention, the model training module 804 can be used to input an image sample dataset into an initial segmentation model and train the initial segmentation model to generate a target segmentation model. Specifically, to train the initial segmentation model constructed in step S203, initial parameters must first be set. Initial parameters may include an initial learning rate lr. base Weight decay parameter w_dacay, iteration count max_iter, initial warmup learning rate warmup_lr base The initial segmentation model is configured with the following parameters: warmup step, learning rate (lr) with exponential decay, and model loss function. The model loss function can include class weights and class region weights. Then, the Adam optimization algorithm is used as the optimizer to update the weight values. Adam is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process; it iteratively updates the neural network weights based on the training data. Finally, the image sample dataset is input into the initial segmentation model for training. The trained initial segmentation model is then saved to obtain the target segmentation model.

[0119] In one specific embodiment, the model training module 804 may also be specifically used for:

[0120] The initial segmentation model is trained by setting parameters including the initial learning rate, weight decay parameter, number of iterations, initial warm-up learning rate, number of warm-up steps, exponential decay strategy of the learning rate, and model loss function; and

[0121] The image sample dataset is input into the initial segmentation model after configuration for training to generate the target segmentation model.

[0122] The image samples in the image sample dataset were subjected to 2x, 4x, ..., 2x magnification. N Subsampling was performed to generate intermediate data C2, C4, ... ;

[0123] For intermediate data Feature extraction is performed through convolution, upsampling is applied using bilinear interpolation, and then compared with intermediate data. Add to generate a feature map ;

[0124] Following the steps of feature extraction via convolution, upsampling via bilinear interpolation, and addition to intermediate data, the feature map is iteratively processed until the feature map O2 is obtained; and

[0125] Bilinear interpolation is performed on the feature map O2 to generate feature maps of the image samples.

[0126] The model testing module 805 is used to input image test data into the target segmentation model for segmentation prediction processing to generate target image data.

[0127] In one embodiment of the present invention, the model testing module 805 can be used to input image test data into a target segmentation model for segmentation prediction processing to generate target image data. Specifically, the image test data collected by the vehicle-mounted front camera is first standardized. In this embodiment, the standardization processing can be performed by resizing and cropping the input image test data to adjust it to a standard size. Then, the standardized image test data is input into the target segmentation model generated in step S204 for segmentation prediction calculation to generate image prediction data. Finally, the image prediction data and the image test data are fused to generate target image data.

[0128] In one specific embodiment, the model testing module 805 may also be specifically used for:

[0129] Image test data is input into the target segmentation model for prediction calculation to generate image prediction data; and

[0130] Image prediction data and image test data are fused together to generate target image data.

[0131] It should be noted that the semantic segmentation device for road scene images provided in the above embodiments and the semantic segmentation method for road scene images provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs its operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the semantic segmentation device for road scene images provided in the above embodiments can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the system into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0132] Embodiments of the present invention also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the semantic segmentation method for road scene images provided in the above embodiments.

[0133] Please refer to Figure 9 As shown, Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention. It should be noted that... Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0134] like Figure 9 As shown, the computer system 900 includes a Central Processing Unit (CPU) 901, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 902 or programs loaded from storage portion 909 into Random Access Memory (RAM) 903. The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An Input / Output (I / O) interface 905 is also connected to the bus 904.

[0135] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card and a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 909 as needed.

[0136] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs various functions defined in the system of the present invention.

[0137] It should be noted that the computer-readable medium shown in the embodiments of the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0140] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the semantic segmentation method for road scene images as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not incorporated into the electronic device.

[0141] Another aspect of the present invention provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the semantic segmentation method for road scene images provided in the various embodiments described above.

[0142] In summary, this invention provides a semantic segmentation method, system, device, and medium for road scene images, applicable to the field of intelligent driving technology. This invention is based on a multi-scale feature fusion network, trained using weighted loss. Furthermore, addressing the lack of categories in semantic segmentation data for road scene images, it employs methods such as segmentation annotation of image training data and data augmentation to expand the database, achieving accurate segmentation prediction of road scene images. In particular, this invention exhibits high segmentation accuracy for small categories and small regions within road scenes.

[0143] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A semantic segmentation method for road scene images, characterized in that, include: Receive an image dataset of a road scene, the image dataset including image training data and image test data; The image training data is preprocessed to generate an image sample dataset; An initial segmentation model is established based on the feature pyramid network. The image sample dataset is input into the initial segmentation model to train the initial segmentation model and generate the target segmentation model. The training employs a loss function that includes class weights and class region weights. The class weights represent the difference between the total number of images in the image sample dataset and the number of images in each class, divided by the total number of images. The class region weights represent the difference between the total number of pixels in the image sample dataset and the number of pixels in each class, divided by the total number of pixels. Satisfy the following formula: in, This represents the total number of images in the image sample dataset. This represents the number of images in each category in the image sample dataset; The category region weight Satisfy the following formula: in, This represents the total number of pixels in the image sample dataset. This represents the number of pixels in each category in the image sample dataset; The loss function satisfies the following formula: in, Represents the model loss function. Indicates category weight, Indicates the category region weight. The loss is calculated from the output of the initial segmentation model; and The image test data is input into the target segmentation model for segmentation prediction processing to generate target image data; The step of inputting the image sample dataset into the initial segmentation model and training the initial segmentation model includes: The image samples in the image sample dataset are subjected to 2x, 4x, ... Multiple downsampling processing to generate intermediate data. , ... ; For the intermediate data Feature extraction is performed through convolution, upsampling is performed using bilinear interpolation, and then compared with the intermediate data. Add to generate a feature map ; The feature map is iteratively processed by following the steps of feature extraction via convolution, upsampling via bilinear interpolation, and addition to the intermediate data, until the feature map is obtained. ;as well as For the feature map Bilinear interpolation is performed to generate a feature map of the image sample, the feature map having the same size as the image sample.

2. The semantic segmentation method for road scene images according to claim 1, characterized in that, The step of inputting the image sample dataset into the initial segmentation model for training to generate the target segmentation model includes: The initial segmentation model is trained with the following parameters: initial learning rate, weight decay parameter, number of iterations, initial warm-up learning rate, number of warm-up steps, exponential decay strategy for the learning rate, and model loss function; and The image sample dataset is input into the initial segmentation model after configuration for training to generate the target segmentation model.

3. The semantic segmentation method for road scene images according to claim 1, characterized in that, The step of preprocessing the image training data to generate an image sample dataset includes: Image samples are extracted from the image training data, the image samples representing road scene images; and The image samples are semantically annotated to generate the image sample dataset.

4. The semantic segmentation method for road scene images according to claim 1, characterized in that, The step of inputting the image test data into the target segmentation model for segmentation prediction processing to generate target image data includes: The image test data is input into the target segmentation model for prediction calculation to generate image prediction data; and The image prediction data and the image test data are fused together to generate the target image data.

5. A semantic segmentation device for road scene images, characterized in that, include: The data receiving module is used to receive an image dataset of a road scene, the image dataset including image training data and image test data; The data preprocessing module is used to preprocess the image training data to generate an image sample dataset; The model building module is used to build an initial segmentation model based on the feature pyramid network; The model training module is used to input the image sample dataset into the initial segmentation model and train the initial segmentation model to generate the target segmentation model. The training employs a loss function including class weights and class region weights. The class weights represent the difference between the total number of images in the image sample dataset and the number of images in each class, divided by the total number of images. The class region weights represent the difference between the total number of pixels in the image sample dataset and the number of pixels in each class, divided by the total number of pixels. Satisfy the following formula: in, This represents the total number of images in the image sample dataset. This represents the number of images in each category in the image sample dataset; The category region weight Satisfy the following formula: in, This represents the total number of pixels in the image sample dataset. This represents the number of pixels in each category in the image sample dataset; The loss function satisfies the following formula: in, Represents the model loss function. Indicates category weight, Indicates the category region weight. The loss is calculated from the output of the initial segmentation model; and The model testing module is used to input the image test data into the target segmentation model for segmentation prediction processing to generate target image data; The model training module is also used to perform multiplication of image samples in the image sample dataset by 2x, 4x, ... Multiple downsampling processing to generate intermediate data. , ... ; For the intermediate data Feature extraction is performed through convolution, upsampling is performed using bilinear interpolation, and then compared with the intermediate data. Add to generate a feature map ; The feature map is iteratively processed by following the steps of feature extraction via convolution, upsampling via bilinear interpolation, and addition to the intermediate data, until the feature map is obtained. ;as well as For the feature map Bilinear interpolation is performed to generate a feature map of the image sample, the feature map having the same size as the image sample.

6. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the semantic segmentation method for road scene images as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the semantic segmentation method for the road scene image according to any one of claims 1 to 4.

Images