A semantic segmentation-based unstructured field road scene recognition method and device

Abstract

The application discloses a kind of unstructured field road scene identification method and device based on semantic segmentation, which comprises the following steps: acquiring unstructured field road scene image construction dataset, and performing semantic labeling on the image dataset;The labeled dataset is augmented and divided;A semantic segmentation model is constructed, a hybrid dilation convolution is integrated into the MobilenetV2 feature extraction network, a channel attention module is introduced to recalibrate the feature channels of each stage of the feature extraction network, and a spatial pyramid pooling module is designed to calculate multi-scale hierarchical features and splice with input features;Initialize the feature extraction network parameters for pre-training, add the trained feature extraction network to the spatial pyramid pooling module and the pixel prediction network, and train on the training set using the stochastic gradient descent method;After training, input the image to be identified into the semantic segmentation model to obtain the segmentation result.This method has good segmentation effect and can achieve balance between accuracy and speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of agricultural field road scene recognition, and particularly relates to a non-structured field road scene recognition method and device based on semantic segmentation. BACKGROUND

[0002] Intelligent agricultural equipment system can autonomously, efficiently and safely complete agricultural operation tasks, and has good operation precision and efficiency. Environmental information perception is one of the key technologies of intelligent agricultural equipment system, which determines the autonomous navigation ability and operation level of agricultural equipment. Machine vision system has the characteristics of wide detection range and rich information acquisition, and is one of the sensing devices mainly used by intelligent agricultural equipment for field information acquisition. Vision-based agricultural field road scene recognition and analysis is an important component of environmental information perception, and the main task is to detect the passable area and identify the static and dynamic obstacle categories, so as to provide a basis for subsequent path planning and decision control. Fast and accurate field road scene recognition is of great significance to ensure the safe and reliable operation of intelligent agricultural equipment in non-structured road environment.

[0003] Traditional methods use artificial design features to recognize and analyze road scenes based on the combination of one or more of surface features such as color, texture and shape. They have good adaptability to structured roads, but lack extraction and expression of deep image features and high-level semantic information, are easily affected by factors such as road morphology, light changes and road interference, and are difficult to apply to complex non-structured field road scene detection.

[0004] Compared with traditional methods, semantic segmentation technology based on deep learning can complete the recognition and analysis of complex image scenes, and has become a mainstream tool for solving high-level visual problems. Semantic segmentation models based on deep convolutional neural networks can realize semantic pixel prediction and classification, and have good segmentation effect, but also have the disadvantages of many weight parameters, high computational complexity, slow inference speed, etc. At the same time, the context information of the image is not fully considered, and the utilization rate of global features is low, resulting in low segmentation accuracy of complex scenes. SUMMARY

[0005] The present application proposes a non-structured field road scene recognition method and device based on semantic segmentation, in order to accurately recognize and analyze the objects contained in the agricultural non-structured road scene in real time. The model uses MobileNetV2 network to extract image features to improve the inference speed; introduces mixed dilated convolution and channel attention module to increase the receptive field of feature maps while preserving more pixel spatial positions and improving the weight coefficients of important feature channels to achieve better prediction; the spatial pyramid pooling module is used to fuse multi-scale pooling features to obtain complete global feature expression and enhance the accuracy of complex road scene recognition.

[0006] In a first aspect, the present application provides a method for identifying an unstructured field road scene based on semantic segmentation, comprising:

[0007] S01, acquiring an unstructured field road scene image to construct a data set, and performing semantic labeling on the image data set;

[0008] S02, data augmentation is performed on the labeled data set, and the augmented data is divided into a training set, a validation set and a test set;

[0009] S03, a semantic segmentation model is constructed based on the Keras deep learning framework, a mixed dilation convolution is integrated into the MobilenetV2 feature extraction network, a channel attention module is introduced to recalibrate the feature channels of each stage of the feature extraction network, and a spatial pyramid pooling module is designed to calculate multi-scale hierarchical features and splice them with the input features;

[0010] S04, the parameters of the feature extraction network are initialized for pre-training, the trained feature extraction network is added to the spatial pyramid pooling module and the pixel prediction network, and the random gradient descent method is used to train on the road image training set;

[0011] S05, after the training is completed, the image to be identified is input into the semantic segmentation model, and the semantic category and position information of the road scene object are output.

[0012] Optionally, in step S03, the semantic segmentation model is constructed based on the Keras deep learning framework, the mixed dilation convolution is integrated into the MobilenetV2 feature extraction network, the channel attention module is introduced to recalibrate the feature channels of each stage of the feature extraction network, and the spatial pyramid pooling module is designed to calculate multi-scale hierarchical features and splice them with the input features, specifically comprising:

[0013] (1) MobileNetV2 is used to extract image feature information, and mixed dilation convolution is integrated into stage 4 and stage 5 of the feature extraction network. The channel attention module is used to fuse features of different stages, and the high-level stage is used to guide the low-level stage to recalibrate the internal feature channels according to the importance;

[0014] (2) A spatial pyramid pooling module is constructed, the input feature information is divided into three levels, the first level uses global average pooling operation to calculate the feature vector, and the second and third levels divide the input features into 2x2 and 4x4 sub-regions, and each sub-region uses average pooling operation to calculate the feature vector;

[0015] (3) The feature vectors calculated by the three different levels are up-sampled and spliced with the input features to obtain global features, and the semantic segmentation image is obtained through convolution, up-sampling and pixel prediction classification.

[0016] Optionally, the step S04 initializes the feature extraction network parameter for pre-training, adds the trained feature extraction network to the spatial pyramid pooling module and the pixel prediction network, and deploys the trained feature extraction network on the road image training set for training by using the stochastic gradient descent method, specifically including:

[0017] (1) The feature extraction network is composed of the MobilnetV2 network, the mixed dilated convolution, and the channel attention module, the parameters are randomly initialized by using the Gaussian distribution with the mean of 0 and the standard deviation of 0.01, the hyperparameters involved in the network training are set, the feature extraction network is deployed on the ImageNet dataset for pre-training by using the cross-entropy loss function as the objective function;

[0018] (2) The trained feature extraction network is removed from the average pooling layer and the classification layer, the spatial pyramid pooling module and the pixel prediction classification network are added, the parameters of the newly added network are randomly initialized by using the Gaussian distribution with the mean of 0 and the standard deviation of 0.01, the hyperparameters involved in the network training are set, and the joint training is performed based on the stochastic gradient descent method by using the multi-task loss function as the objective function.

[0019] In a second aspect, the present application further provides a non-structured field road scene recognition device based on semantic segmentation, comprising: an image acquisition and labeling module, configured to acquire a non-structured field road scene image construction dataset, and perform semantic labeling on the image dataset;

[0020] A data augmentation classification module is configured to perform data augmentation on the labeled dataset, and the augmented data is divided into a training set, a validation set, and a test set;

[0021] A semantic segmentation model construction module is configured to construct a semantic segmentation model based on a Keras deep learning framework, integrate a mixed dilated convolution into a MobilenetV2 feature extraction network, introduce a channel attention module to recalibrate the feature channels of each stage of the feature extraction network, and design a spatial pyramid pooling module to calculate multi-scale hierarchical features and splice the input features;

[0022] A semantic segmentation model training module is configured to initialize the feature extraction network parameter for pre-training, add the trained feature extraction network to the spatial pyramid pooling module and the pixel prediction network, and deploy the trained feature extraction network on the road image training set for training by using the stochastic gradient descent method;

[0023] A segmentation result output module is configured to input a to-be-identified image to the semantic segmentation model after training, and output semantic categories and position information of road scene objects.

[0024] Optionally, the semantic segmentation model construction module specifically includes:

[0025] The feature extraction unit is used for extracting image feature information by adopting a MobileNetV2 front 5-stage convolutional neural network, fusing a mixed dilated convolution into stages 4 and 5 of the feature extraction network, fusing different stage features by adopting a channel attention module, and guiding internal feature channels according to importance by using strong semantic information of a high-level stage; the spatial pyramid pooling unit is used for constructing a spatial pyramid pooling module, dividing input feature information into 3 levels, calculating a feature vector by adopting a global average pooling operation in the first level, dividing input features into 2x2 and 4x4 sub-regions in the second and third levels, and calculating a feature vector by adopting an average pooling operation in each sub-region;

[0026] The prediction segmentation unit is used for upsampling the feature vectors calculated by the 3 different levels and splicing the input features to obtain a global feature, performing convolution, upsampling, and pixel prediction classification to obtain a semantic segmentation image.

[0027] Optionally, the semantic segmentation model training module specifically comprises:

[0028] The pre-training unit is used for constructing a feature extraction network by adopting a MobilnetV2 network, a mixed dilated convolution, and a channel attention module, randomly initializing parameters by adopting a Gaussian distribution with a mean of 0 and a standard deviation of 0.01, setting hyperparameters involved in network training, and pre-training the feature extraction network on an ImageNet dataset by deploying the feature extraction network on the ImageNet dataset as a target function by using a cross-entropy loss function; the joint training unit is used for removing an average pooling layer and a classification layer from the trained feature extraction network, adding a spatial pyramid pooling module and a pixel prediction classification network, randomly initializing parameters of the newly added network by adopting a Gaussian distribution with a mean of 0 and a standard deviation of 0.01, setting hyperparameters involved in network training, and performing joint training based on a stochastic gradient descent method by using a multi-task loss function as a target function.

[0029] According to the above technical solution, the application provides a non-structured field road scene recognition method and device based on semantic segmentation, which has the following advantages:

[0030] ①A MobileNetV2 light-weight convolutional neural network is adopted to extract image features to improve the image feature extraction speed, a mixed dilated convolution is fused into the feature extraction network, the receptive field is increased on the basis of ensuring the feature map resolution, and the continuity and integrity of information are maintained;

[0031] ②A channel attention module is introduced to recalibrate feature channel of each stage of the feature extraction network according to importance, improve feature utilization efficiency and prediction consistency;

[0032] ③Design space pyramid pooling module calculates multi-scale hierarchical features and splices and fuses with input features to obtain more effective global scene context information and enhance the accuracy of complex road scene recognition;

[0033] ④The unstructured field road scene recognition method and device based on semantic segmentation have the advantages of high accuracy, fast reasoning speed and small parameter quantity, and can better balance the accuracy and speed. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A flowchart of an unstructured field road scene recognition method based on semantic segmentation is provided for the embodiments of the present application.

[0035] Figure 2 A schematic diagram of a feature extraction network fusion mixed dilated convolution structure is provided for the embodiments of the present application.

[0036] Figure 3 A schematic diagram of a channel attention module structure is provided for the embodiments of the present application.

[0037] Figure 4 A schematic diagram of an unstructured field road scene recognition model structure based on semantic segmentation is provided for the embodiments of the present application.

[0038] Figure 5 A schematic diagram of an unstructured field road scene recognition device structure based on semantic segmentation is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0039] The specific embodiments of the application will be further described below in conjunction with the drawings, and the following embodiments are only used to more clearly illustrate the technical solutions of the present application and cannot be used to limit the protection scope of the present application.

[0040] Figure 1 A flowchart of an unstructured field road scene recognition method based on semantic segmentation is provided for the embodiments of the present application, as shown in Figure 1 The method comprises the following steps:

[0041] 101, acquire an unstructured field road scene image construction dataset, and perform semantic labeling on the image dataset;

[0042] 102, perform data augmentation on the labeled dataset, and divide the augmented data into a training set, a validation set and a test set;

[0043] 103. Construct a semantic segmentation model based on the Keras deep learning framework, integrate hybrid dilated convolution into the MobilenetV2 feature extraction network, introduce a channel attention module to recalibrate the feature channels at each stage of the feature extraction network, and design a spatial pyramid pooling module to calculate multi-scale hierarchical features and concatenate them with the input features.

[0044] 104. Initialize the parameters of the feature extraction network for pre-training, add the trained feature extraction network to the spatial pyramid pooling module and the pixel prediction network, and deploy it on the road image training set for training using stochastic gradient descent; 105. After training, input the image to be recognized into the semantic segmentation model and output the semantic category and location information of the road scene object.

[0045] Step 101 includes the following specific steps:

[0046] (1.1) Position the camera at a downward angle of 20° to 30° to the horizontal ground, with a height of about 150cm above the ground, and collect images of field roads under different weather conditions, different light intensities, and different environmental backgrounds to construct an image dataset;

[0047] (1.2) Use annotation tools to label the location and category of the objects to be trained in the image dataset;

[0048] In this embodiment, Labelme software is used for image annotation. Open Labelme software, click "Open Dir," and select the directory containing the images to be annotated. Use "Create Polygons" to plot points on the target object, forming a closed region with its ends connected. In the pop-up Labelme window, enter the label name and click "OK" to complete the annotation. Save the annotated image using "Save," and click the "Next Image" option to annotate the next image until all images are annotated. The annotated images will generate a .json file with the same name in the image directory. Use the Labelme batch conversion tool to convert the .json file into a .png format label image.

[0049] Step 102 includes the following specific steps:

[0050] (2.1) The labeled image dataset is augmented through geometric and color transformations;

[0051] This embodiment uses Python programming for data augmentation. The original image and its corresponding label image are placed in two separate folders and subjected to the same data augmentation operations. Geometric transformations include horizontal flipping, vertical flipping, image rotation, and image scaling. The image rotation angles are selected as -45°, -30°, -15°, 45°, 30°, and 15°, and the image scaling scales are selected as 0.8x, 0.9x, 1.1x, and 1.2x. Color transformations involve adjusting the image contrast, brightness, and chroma. The contrast adjustment values ​​are 0.8x, 0.9x, 1.1x, and 1.2x of the original value; the brightness adjustment values ​​are 0.85x, 0.95x, 1.05x, and 1.15x of the original value; and the chroma adjustment values ​​are 0.7x, 0.8x, 0.9x, 1.1x, 1.2x, and 1.3x of the original value.

[0052] (2.2) The augmented dataset is randomly split into training set, validation set and test set in a ratio of 8:1:1.

[0053] Step 103 includes the following specific steps:

[0054] (3.1) Image feature information extraction is performed using the first 5 stages of the MobileNetV2 convolutional neural network. Hybrid dilated convolutions are integrated into stages 4 and 5 of the feature extraction network. A channel attention module is used to fuse features from different stages. The strong semantic information of the higher stages guides the lower stages to relabel the internal feature channels according to their importance. The specific process includes: ① In this embodiment, the deep learning framework is Keras, and the program is designed based on Python on the Windows 10 operating system. The specific design idea is: the first 5 stages of the MobileNetV2 convolutional neural network are used to extract image feature information. After 4 downsampling operations, the final output feature map size is 1 / 16 of the input image. Among them, stage 5 expands the number of feature channels based on stage 4 without performing downsampling operations. A set of hybrid dilated convolutions consists of n (n≥2) K×K (K≥2) convolutional kernels, and the dilation rate of the convolutional kernels is set to [r1,…r n Each layer uses a different dilation rate, and multiple consecutive dilation convolution operations with different dilation rates can avoid problems such as spatial gaps and information discontinuities in the receptive field. Furthermore, the aforementioned dilation rates [r1,…r]... n The element values ​​in the array must increase gradually in a step-like manner, cannot have a common divisor greater than 1, and must satisfy the following formula constraints:

[0055] M i =max[M i+1 -2r i M i+1 -2(M i+1 -ri ),r i ]

[0056] M2≤K

[0057] In the formula, K represents the convolution kernel size, r i M represents the dilation rate of the i-th convolutional kernel. i M represents the dilation rate calculated by the i-th convolutional kernel. M2≤K can guarantee that there is no grid problem in the hybrid dilated convolution operation.

[0058] Hybrid dilated convolutions are incorporated into stages 4 and 5 of the feature extraction network. In stage 4, the six inverted residuals and linear bottlenecks blocks (IRLBBs) are divided into two groups of hybrid dilated convolutional units, with the depthwise separable convolution dilation rates in each group set to 1, 2, and 3. In stage 5, the first three of the four inverted residuals and linear bottlenecks blocks are selected as one group, with dilation rates set to 1, 2, and 3. Figure 2 This is a schematic diagram of the hybrid dilated convolution fusion feature extraction network structure. IRLBB is the linear bottleneck inverse residual module, N is the number of times the linear bottleneck inverse residual module is executed, and HDC(1,2,3) represents the hybrid dilated convolution (HDC) with dilation rates of 1, 2, and 3.

[0059] ②In this embodiment, the channel attention module is composed as follows: Figure 3 As shown, the high-level stage and the low-level stage are concatenated by channels. The importance of each channel is obtained through the weight learning module, and a new weight vector is generated. The weights of the feature channels in the low-level stage are recalibrated in a multiplicative manner to increase the weights of important features and decrease the weights of unimportant features, thereby enhancing the discriminative features of the entire stage. Figure 3 In this context, Global pooling is the global average pooling operation, which compresses the feature map into a one-dimensional vector. Sigmoid is the activation function, ReLU is the non-linear activation function, and 1×1Conv is the projective convolution.

[0060] (3.2) Construct a spatial pyramid pooling module to divide the input feature information into three levels. The first level uses global average pooling to calculate the feature vector. The second and third levels divide the input features into 2×2 and 4×4 sub-regions, respectively, and each sub-region uses average pooling to calculate the feature vector. The specific process includes:

[0061] Figure 4The spatial pyramid pooling module includes three layers of features at different scales. The first layer uses global average pooling to calculate feature vectors and obtain global contextual information at the image level. The second and third layers divide the input feature map into sub-regions of different sizes, and use average pooling to calculate feature vectors in each sub-region as local contextual information for all pixels within that sub-region. To ensure consistent feature weights across all layers, a 1×1 projective convolution is used to adjust the pooling feature channels of each layer to 1 / S of the input feature channels, where S represents the pyramid dimension. In this embodiment, the spatial pyramid dimension S is equal to 3, and the number of sub-regions divided in the input feature map of layers 1, 2, and 3 are 1, 4, and 16, respectively.

[0062] (3.3) Upsample the feature vectors calculated at the three different levels and concatenate them with the input features to obtain global features. After convolution, upsampling, and pixel prediction classification, a semantic segmentation image is obtained. The specific process includes:

[0063] The bilinear interpolation algorithm is used to upsample the pooling features at each level to restore them to the same size as the input feature map. Then, the features at different levels are concatenated with the input features to obtain the global features. Finally, 3x3 convolution and 1x1 convolution are used to integrate features and adjust channels. After upsampling and pixel classification operations, the final prediction map is generated. Figure 4 This is a schematic diagram of the structure of an unstructured field road scene recognition model based on semantic segmentation. In the diagram, CAB represents the channel attention block (CAB), HDC(1,2,3) represents the hybrid dilated convolution (HDC) with dilation rates of 1, 2, and 3, and Softmax is the multi-classification function.

[0064] Step 104 includes the following specific steps:

[0065] (4.1) A feature extraction network is constructed using the MobilnetV2 network, hybrid dilated convolution, and channel attention modules. Parameters are randomly initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. Hyperparameters involved in network training are set. The feature extraction network is pre-trained on the ImageNet dataset using the cross-entropy loss function as the objective function. The specific process includes:

[0066] ① To improve training speed and efficiency, ImageNet data images are converted to TFRecord format. Each field of the TFRecord file records information such as image name, dimensions, encoding data, and label definition.

[0067] ② The network parameters were randomly initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. The weight decay coefficient was 0.0005, the batch size was set to 32, the initial learning rate was 0.025, and the momentum factor was 0.9. During the initial training phase, a learning rate warm-up was performed. In the first 1000 batches, the learning rate was linearly increased from 0 to 0.025. Subsequently, the learning rate decayed in a piecewise constant manner with increasing iteration count.

[0068] (4.2) Remove the average pooling layer and classification layer from the trained feature extraction network, and add a spatial pyramid pooling module and a pixel prediction classification network. Randomly initialize the parameters of the newly added network using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. Set the hyperparameters involved in network training, and perform joint training using the multi-task loss function as the objective function based on stochastic gradient descent. The specific process includes:

[0069] ① Remove the average pooling layer and classification layer from the pre-trained feature extraction network, and add a spatial pyramid pooling module and a pixel prediction classification network. Freeze the first four stages of the convolutional layers of the feature extraction network. Randomly initialize the parameters of the newly added network using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. Set the batch size to 8, the momentum factor to 0.9, the learning rate to 0.001, the decay coefficient to 0.8, and the number of training epochs to 50. After the iteration is completed, unfreeze the first four stages of the feature extraction network and train the entire model. The learning rate is 0.0001, the decay coefficient is 0.5, and the number of training epochs remains 50.

[0070] ② During training, a multi-task loss function is used to predict and classify the target object pixels. This multi-task loss function consists of cross-entropy loss and Dice coefficient loss. Cross-entropy loss calculates the cross-entropy between the one-hot encoded vector and the predicted value for each pixel, then sums and averages the results over all pixels to measure the difference between the predicted and true pixel categories. A smaller loss indicates higher prediction accuracy. Dice coefficient loss is a set similarity measure used to calculate the pixel similarity between the real and predicted segmented images, with a value range of [0,1]. The multi-task loss function is defined as follows:

[0071]

[0072] In the formula, L ce For cross-entropy loss, L dice The loss function is the Dice coefficient, where N is the total number of pixels and T is the number of classes. Let c be the one-hot encoded vector of the label. It is set to 1 when the category of pixel i matches the category of c, and 0 otherwise. Let X represent the probability that pixel i is predicted to be of category c, where X is the set of pixels in the true segmented image and Y is the set of pixels in the predicted segmented image.

[0073] Step 105 includes the following specific steps:

[0074] The image to be identified is input into a trained semantic segmentation model to obtain the semantic segmentation result of the target object in the unstructured field road scene. The segmentation result includes the target object category and location information.

[0075] Figure 5 A schematic diagram of a semantic segmentation-based unstructured field road scene recognition device is provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the device includes:

[0076] The image acquisition and annotation module 501 is used to acquire unstructured field road scene images to construct a dataset and perform semantic annotation on the image dataset;

[0077] The data augmentation and classification module 502 is used to augment the labeled dataset, and the augmented data is divided into a training set, a validation set, and a test set.

[0078] The semantic segmentation model building module 503 is used to build a semantic segmentation model based on the Keras deep learning framework. It incorporates hybrid dilated convolution into the MobilenetV2 feature extraction network, introduces a channel attention module to recalibrate the feature channels at each stage of the feature extraction network, and designs a spatial pyramid pooling module to calculate multi-scale hierarchical features and concatenate them with the input features.

[0079] The semantic segmentation model training module 504 is used to initialize the parameters of the feature extraction network for pre-training, add the trained feature extraction network to the spatial pyramid pooling module and the pixel prediction network, and deploy it on the road image training set for training using stochastic gradient descent.

[0080] The segmentation result output module 505 is used to input the image to be recognized into the semantic segmentation model after training and output the semantic category and location information of the road scene object.

[0081] The image acquisition and annotation module 501 specifically includes:

[0082] The image acquisition unit is used to make the camera at a downward angle of 20° to 30° to the horizontal ground, about 150cm above the ground, to collect images of field roads under different weather conditions, different light intensities, and different environmental backgrounds to construct an image dataset.

[0083] The image annotation unit is used to label the location and category of the objects to be trained in the image dataset using annotation tools.

[0084] The data augmentation and classification module 502 specifically includes:

[0085] The data augmentation unit is used to augment the labeled image dataset through geometric and color transformations.

[0086] The data classification unit is used to randomly split the augmented dataset into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0087] The semantic segmentation model construction module 503 specifically includes:

[0088] The feature extraction unit uses the first five stages of the MobileNetV2 convolutional neural network to extract image feature information. It integrates hybrid dilated convolution into stages 4 and 5 of the feature extraction network and uses a channel attention module to fuse features from different stages. It uses strong semantic information from higher stages to guide lower stages to recalibrate internal feature channels according to their importance. The spatial pyramid pooling unit is used to construct the spatial pyramid pooling module. It divides the input feature information into three levels. The first level uses global average pooling to calculate the feature vector. The second and third levels divide the input features into 2×2 and 4×4 sub-regions, respectively. Each sub-region uses average pooling to calculate the feature vector.

[0089] The prediction segmentation unit is used to upsample the feature vectors calculated at the three different levels and concatenate them with the input features to obtain global features. After convolution, upsampling and pixel prediction classification, a semantic segmentation image is obtained.

[0090] The semantic segmentation model training module 504 specifically includes:

[0091] The pre-training unit is used to construct a feature extraction network from the MobilnetV2 network, hybrid dilated convolution, and channel attention modules. Parameters are randomly initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. Hyperparameters involved in network training are set, and the feature extraction network is deployed on the ImageNet dataset for pre-training using the cross-entropy loss function as the objective function. The joint training unit removes the average pooling layer and classification layer from the trained feature extraction network and adds a spatial pyramid pooling module and a pixel prediction classification network. Parameters of the newly added network are randomly initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01. Hyperparameters involved in network training are set, and joint training is performed using a multi-task loss function as the objective function based on stochastic gradient descent.

[0092] The segmentation result output module 505 specifically includes:

[0093] The image to be identified is input into a trained semantic segmentation model to obtain the semantic segmentation result of the target object in the unstructured field road scene. The segmentation result includes the target object category and location information.

[0094] The method and apparatus of the present invention are one-to-one correspondences. Therefore, the calculation process of some parameters in the method is also applicable to the calculation process in the apparatus, and will not be described in detail in the apparatus.

[0095] Numerous specific details are set forth in this specification. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the inventive format examples, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for recognizing unstructured field road scenes based on semantic segmentation, characterized in that, include: S01. Obtain unstructured field road scene images to construct a dataset, and perform semantic annotation on the image dataset; S02. Augment the labeled dataset and divide the augmented data into training set, validation set, and test set. S03. Construct a semantic segmentation model based on the Keras deep learning framework and integrate it into the MobilenetV2 feature extraction network. Hybrid dilated convolution is used, and a channel attention module is introduced to recalibrate the feature channels at each stage of the feature extraction network. A spatial pyramid pooling module is designed to calculate multi-scale hierarchical features and concatenate them with the input features. Step S03 specifically includes: (1) The first five stages of the MobileNetV2 convolutional neural network are used to extract image feature information. Hybrid dilated convolution is integrated into stages 4 and 5 of the feature extraction network. The channel attention module is used to fuse features from different stages. The strong semantic information of the higher stages is used to guide the lower stages to recalibrate the internal feature channels according to their importance. (2) Construct a spatial pyramid pooling module to divide the input feature information into three levels. The first level uses global average pooling to calculate the feature vector. The second and third levels divide the input features into 2×2 and 4×4 sub-regions, respectively. The feature vector of each sub-region is calculated using average pooling. (3) Upsample the feature vectors calculated at three different levels and concatenate them with the input features to obtain global features. After convolution, upsampling and pixel prediction classification, a semantic segmentation image is obtained. S04. Initialize the feature extraction network parameters for pre-training, and then add the trained feature extraction network to the spatial gold... The pyramid pooling module and pixel prediction network are deployed on a road image training set and trained using stochastic gradient descent. S05. After training, the image to be recognized is input into the semantic segmentation model, which outputs the semantic category and location information of the road scene object.

2. The method for recognizing unstructured field road scenes based on semantic segmentation according to claim 1, characterized in that, Step S04 specifically includes: (1) The feature extraction network is constructed by MobilnetV2 network, hybrid dilated convolution and channel attention module. The parameters are randomly initialized by Gaussian distribution with mean of 0 and standard deviation of 0.

01. The hyperparameters involved in network training are set. The feature extraction network is deployed on ImageNet dataset for pre-training using cross-entropy loss function as objective function. (2) Remove the average pooling layer and classification layer from the trained feature extraction network, add a spatial pyramid pooling module and a pixel prediction classification network, randomly initialize the parameters of the newly added network using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01, set the hyperparameters involved in network training, and perform joint training based on stochastic gradient descent using the multi-task loss function as the objective function.

3. A device for recognizing unstructured field road scenes based on semantic segmentation, characterized in that, include: The image acquisition and annotation module is used to acquire unstructured field road scene images to construct a dataset and perform semantic annotation on the image dataset; The data augmentation and classification module is used to augment the labeled dataset, and the augmented data is divided into training set, validation set and test set; The semantic segmentation model building module is used to build semantic segmentation models based on the Keras deep learning framework, in MobileNetV2. Hybrid dilated convolution is incorporated into the feature extraction network, a channel attention module is introduced to recalibrate the feature channels at each stage of the feature extraction network, and a spatial pyramid pooling module is designed to calculate multi-scale hierarchical features and concatenate them with the input features. The semantic segmentation model training module is used to initialize the parameters of the feature extraction network for pre-training, add the trained feature extraction network to the spatial pyramid pooling module and the pixel prediction network, and deploy it on the road image training set for training using stochastic gradient descent. The segmentation result output module is used to input the image to be recognized into the semantic segmentation model after training and output the semantic category and location information of the road scene object; The semantic segmentation model construction module specifically includes: The feature extraction unit is used to extract image feature information using the first 5 stages of the MobileNetV2 convolutional neural network. Hybrid dilated convolution is integrated into stages 4 and 5 of the feature extraction network. The channel attention module is used to fuse features from different stages. The strong semantic information of the higher stages is used to guide the lower stages to recalibrate the internal feature channels according to their importance. The spatial pyramid pooling unit is used to construct the spatial pyramid pooling module. It divides the input feature information into three levels. The first level uses global average pooling to calculate the feature vector. The second and third levels divide the input features into 2×2 and 4×4 sub-regions, respectively. The feature vector of each sub-region is calculated using average pooling. The prediction segmentation unit is used to upsample the feature vectors calculated at three different levels and concatenate them with the input features to obtain global features. After convolution, upsampling, and pixel prediction classification, a semantic segmentation image is obtained.

4. The unstructured field road scene recognition device based on semantic segmentation according to claim 3, characterized in that, The semantic segmentation model training module specifically includes: The pre-training unit is used to construct a feature extraction network from the MobilnetV2 network, hybrid dilated convolution, and channel attention module. The parameters are randomly initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.

01. The hyperparameters involved in network training are set, and the feature extraction network is deployed on the ImageNet dataset for pre-training using the cross-entropy loss function as the objective function. The joint training unit is used to remove the average pooling layer and classification layer from the trained feature extraction network, add a spatial pyramid pooling module and a pixel prediction classification network, randomly initialize the parameters of the newly added network using a Gaussian distribution with a mean of 0 and a standard deviation of 0.01, set the hyperparameters involved in network training, and perform joint training based on stochastic gradient descent using a multi-task loss function as the objective function.

Images