Semantic segmentation method and device based on multi-scale cavity convolutional neural network and multistage conditional random field

Abstract

The invention provides a semantic segmentation method and device based on a multi-scale cavity convolutional neural network and a multistage conditional random field, and the method comprises the steps: firstly, carrying out the network layer construction of the multi-scale cavity convolutional neural network based on a residual network, and generating a multi-scale cavity convolutional neural network model; afterwards, on the basis of the model, feature extraction is carried out on an original image, and a probability prediction map is generated; then, on the basis of the probability prediction map, a potential energy function is constructed, and a multilevel conditional random field potential energy function is generated; and finally, a combined potential energy function is generated by setting constraint conditions of the multilevel conditional random field potential energy function, and semantic segmentation prediction of the original image is completed by solving an extreme value of the combined potential energy function. According to the embodiment of the invention, the method achieves the construction of a lightweight mACNN model, guarantees the reduction of calculation resource consumption, improves the segmentation performance, and improves the accuracy of edge refining and semantic segmentation through the MLCRF optimization combined with the edge features.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a semantic segmentation method and device based on a multi-scale hole convolutional neural network and a multi-level conditional random field. Background Art

[0002] Image segmentation is one of the core tasks in computer vision. It is widely used in many fields such as autonomous driving, medical image analysis, industrial quality inspection, intelligent video surveillance, environmental monitoring, etc. It has important research value and practical significance. With the rapid development of deep learning, semantic segmentation technology based on deep learning has gradually become mainstream and has made significant progress. However, although its accuracy has stabilized, the semantic segmentation task still faces many challenges. For example, how to deal with spatial context information in complex images, and how to balance the accuracy and computational complexity of the model.

[0003] Existing semantic segmentation technologies are mainly divided into two categories: traditional machine learning methods and deep learning methods. Traditional machine learning methods, such as graph cut algorithms and support vector machines, have advantages in interpretability and low requirements for data volume and computing resources, but they perform poorly when processing complex patterns and large-scale data. Deep learning methods, such as FCN and U-Net, outperform traditional methods in performance, especially on large-scale data sets, and can automatically learn features and capture multi-scale information, but they require a large amount of labeled data and computing resources, and are usually lacking in interpretability due to the complexity of the model. In addition, deep learning models are susceptible to the vanishing gradient problem, which makes model training and reasoning face great challenges.

[0004] Therefore, one or more methods are needed to solve the above problems.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present application shows a semantic segmentation method and device based on a multi-scale dilated convolutional neural network and a multi-level conditional random field.

[0007] In the first aspect, the present application shows a semantic segmentation method based on a multi-scale dilated convolutional neural network and a multi-level conditional random field, the method comprising:

[0008] Based on the residual network, a multi-scale atrous convolutional neural network model is generated by constructing the network layer of the multi-scale convolutional neural network.

[0009] Based on the multi-scale hole convolutional neural network model, a probability prediction graph is generated by extracting features from the original image;

[0010] Based on the probability prediction graph, a multi-level conditional random field potential energy function is generated by constructing a potential energy function of the multi-level conditional random field model;

[0011] Based on the image segmentation performance, the constraints of the multi-level conditional random field potential energy function are set to generate a combined potential energy function, and the extreme value of the combined potential energy function is solved to complete the semantic segmentation prediction of the original image.

[0012] In an exemplary embodiment of the present disclosure, the multi-scale dilated convolutional neural network model includes an input layer, a dilated convolutional layer, a pooling layer, and a deconvolutional layer, wherein:

[0013] Based on the residual network, the number of convolution kernel layers is skipped by a progressive mechanism to generate a hole convolution layer;

[0014] Based on the spatial structure information of the image, the deconvolution layer is generated by replacing the fully connected layer.

[0015] In an exemplary embodiment of the present disclosure, feature extraction is performed on the original image, including:

[0016] Based on the nearest neighbor interpolation method, the original image is resized through the input layer to generate a standard image;

[0017] Based on a preset convolution kernel size, the standard image is captured using the dilated convolution layer to generate feature structure information;

[0018] Based on the feature structure information, a fused feature map is generated by adaptively adjusting the layer parameters of the pooling.

[0019] In an exemplary embodiment of the present disclosure, feature extraction is performed on the original image, including:

[0020] Based on the number of predetermined pixel categories, the number of feature channels is generated by performing a convolution operation on the number of channels;

[0021] Based on the number of feature channels, the intermediate image of the fused feature map is multiplied by upsampling through the deconvolution layer to generate a probability prediction map.

[0022] In an exemplary embodiment of the present disclosure, a potential energy function of a multi-level conditional random field model is constructed, including:

[0023] Based on the edge detection algorithm, the edge potential energy function is generated by extracting the contour feature information of the original image;

[0024] Based on a simple linear iterative clustering algorithm, a superpixel segmentation map is generated by performing superpixel segmentation on the original image;

[0025] Based on the superpixel segmentation map, generating a superpixel probability map by converting the probability prediction map;

[0026] Based on the robust Potts model, high-dimensional complex feature label values ​​are generated by adding linear truncation function labels to the high-dimensional complex features in the image.

[0027] In an exemplary embodiment of the present disclosure, a potential energy function of a multi-level conditional random field model is constructed, including:

[0028] Based on the superpixel probability map, a superpixel unary potential energy function is generated by defining a superpixel level unary potential energy function of a multi-level conditional random field model;

[0029] Based on the high-dimensional complex feature label value, a superpixel binary potential function is generated by extracting superpixel level features of the superpixel segmentation map;

[0030] Based on the high-dimensional complex feature label value, a pixel unary potential energy function is generated by modeling the pixel-level features of the probability prediction map;

[0031] Based on the high-dimensional complex feature label value, by extracting the pixel-level features of the original image, a pixel binary potential energy function is generated;

[0032] By integrating the superpixel unary potential energy function, the superpixel binary potential energy function, the pixel unary potential energy function, and the pixel binary potential energy function, a multi-level conditional random field potential energy function is generated.

[0033] In an exemplary embodiment of the present disclosure, by setting the constraint conditions of the multi-level conditional random field potential energy function, generating a combined potential energy function, and solving the extreme value of the combined potential energy function, the method includes:

[0034] Based on the edge potential energy function, generating a pixel multivariate potential energy function by performing edge constraints on the pixel unary potential energy function and the pixel binary potential energy function;

[0035] Based on the pixel multivariate potential energy function, generating a combined potential energy function by performing segmentation constraints on the superpixel unary potential energy function and the superpixel binary potential energy function;

[0036] Based on the image segmentation performance, the semantic segmentation prediction of the original image is completed by solving the extreme points of the combined potential energy function.

[0037] In the second aspect, the present application shows a semantic segmentation device based on a multi-scale hole convolutional neural network and a multi-level conditional random field, the device comprising:

[0038] Multi-scale hole convolutional neural network model construction module, used to construct network layers of multi-scale convolutional neural networks;

[0039] An image feature extraction module is used to extract features from the original image;

[0040] Potential energy function construction module, used to construct the potential energy function of the multi-level conditional random field model;

[0041] The semantic segmentation module is used to set the constraint conditions of the multi-level conditional random field potential energy function, generate a combined potential energy function, and solve the extreme value of the combined potential energy function.

[0042] In a third aspect, the present application shows an electronic device, which includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in any of the above aspects.

[0043] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a method as described in any of the above aspects.

[0044] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any of the above aspects.

[0045] The technical solution provided by this application may have the following beneficial effects:

[0046] Based on the embodiment of the present disclosure, first, according to the residual network, a multi-scale hole convolutional neural network model is generated by constructing a network layer of the multi-scale convolutional neural network. After that, based on the model, a probability prediction graph is generated by extracting features from the original image. Then, based on the probability prediction graph, a multi-level conditional random field potential function is generated by constructing a potential energy function. Finally, by setting the constraints of the multi-level conditional random field potential function, a combined potential energy function is generated, and by solving the extreme value of the combined potential energy function, the semantic segmentation prediction of the original image is completed. Therefore, the embodiment of the present disclosure provides a semantic segmentation framework model based on mACNN-MLCRF, which can effectively combine the advantages of convolutional neural networks and conditional random fields, and improve the prediction accuracy of the model. While reducing the amount of calculation, efficient segmentation performance is achieved, ensuring more accurate edge refinement and capture of global context information, and improving the overall segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of the steps of a semantic segmentation method based on a multi-scale hole convolutional neural network and a multi-level conditional random field in the present application.

[0048] Figure 2 This is a decision logic flow chart of a semantic segmentation method based on a multi-scale hole convolutional neural network and a multi-level conditional random field in the present application.

[0049] Figure 3 This is a multi-scale atrous convolutional neural network model diagram of a semantic segmentation method based on a multi-scale atrous convolutional neural network and a multi-level conditional random field in the present application.

[0050] Figure 4 This is a deconvolution layer upsampling processing flowchart of a semantic segmentation method based on a multi-scale hole convolutional neural network and a multi-level conditional random field in the present application.

[0051] Figure 5 This is a structural block diagram of a semantic segmentation device based on a multi-scale hole convolutional neural network and a multi-level conditional random field in the present application.

[0052] Figure 6 It is a block diagram of an electronic device of the present application.

[0053] Figure 7 It is a block diagram of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] Reference Figure 1 , shows a flowchart of the steps of a semantic segmentation method based on a multi-scale hole convolutional neural network and a multi-level conditional random field of the present application, which can be applied to electronic devices, wherein the method can specifically include the following steps:

[0056] Step S101, based on the residual network, by constructing the network layer of the multi-scale convolutional neural network, a multi-scale hole convolutional neural network model is generated;

[0057] Step S102, generating a probability prediction graph by extracting features from the original image based on the multi-scale atrous convolutional neural network model;

[0058] Step S103, based on the probability prediction graph, generating a multi-level conditional random field potential energy function by constructing a potential energy function of the multi-level conditional random field model;

[0059] Step S104, based on the image segmentation performance, by setting the constraint conditions of the multi-level conditional random field potential energy function, generating a combined potential energy function, and by solving the extreme value of the combined potential energy function, completing the semantic segmentation prediction of the original image.

[0060] Below, as Figure 2 As shown, a semantic segmentation method based on a multi-scale hole convolutional neural network and a multi-level conditional random field in an embodiment of the present disclosure will be further explained.

[0061] In step S101, a multi-scale atrous convolutional neural network model can be generated by constructing a network layer of a multi-scale convolutional neural network based on a residual network.

[0062] In some optional embodiments of this example, a lightweight multi-scale convolutional neural network (Modified Multiscale Atrous CNN, mACNN) integrated with dilated convolution is first designed. The network layers of the multi-scale dilated convolutional neural network model include an input layer, a dilated convolution layer, a pooling layer, and a deconvolution layer, wherein the input layer and the pooling layer are the same as the traditional multi-scale convolutional neural network.

[0063] The difference is that, Figure 3 As shown in Figure 1, the idea of ​​"shortcut connections" in the residual network is first introduced into the multi-scale convolutional neural network. Specifically, by setting an "invalid layer" and using the "shortcut" mechanism to skip some convolutional layers, a hollow convolutional layer is constructed.

[0064] The benefit of this setting is that as the size of the convolution kernel increases, the model can skip more convolution layers through "invalid layers", making it easier to strike a balance between large-scale and small-scale features.

[0065] At the same time, to ensure that the output feature maps of different convolutional layers have the same size when performing multi-scale feature fusion, we can adaptively adjust the parameters of the pooling layer in the next layer of the network after the dilated convolution layer.

[0066] In addition, unlike the traditional convolution operation, which slides the filter (convolution kernel) on the input data and calculates the output value at each position, the dilated convolution introduces intervals between the filters, so that the filter slides on the input data at a certain distance instead of densely sliding. The dilated convolution can significantly increase the receptive field and help capture features of different scales, thereby improving segmentation accuracy. At the same time, since the dilated convolution does not increase the number of parameters, it can reduce the computational burden of the model to a certain extent.

[0067] However, in the process of feature extraction, the continuous convolution operation will gradually reduce the size of the feature map. However, in the image segmentation task, the output of the model should be a feature map of the same size as the input image. Therefore, the second difference between this example and the traditional multi-scale convolutional neural network is that the fully connected network at the end of the model is replaced with a deconvolution layer for upsampling.

[0068] In step S102, a probability prediction graph can be generated by extracting features from the original image based on the multi-scale atrous convolutional neural network model.

[0069] In some optional embodiments of this example, the constructed multi-scale dilated convolutional neural network model is used to process the input data. First, the input data is received through the input layer, and the image (i.e., the original image) given by the data set is resized to a standard image of the same size using the nearest neighbor interpolation method.

[0070] Subsequently, the designed network uses four different sizes of convolution kernels, namely 1×1, 3×3, 5×5, and 7×7, to capture multi-scale features. The small-scale convolution kernels can capture more detailed information, while the large-scale convolution kernels can extract higher-level semantic information. The combination of the two can obtain more comprehensive feature structure information.

[0071] The features are then pooled through the adaptively adjusted pooling layer, and the feature maps after pooling at each scale are superimposed to generate a fused feature map, which is sent to the deconvolution layer for feature extraction.

[0072] In some optional embodiments of this example, before deconvolution processing is performed, the number of feature channels of the output probability prediction map is limited by a convolution operation according to the predetermined number of pixel categories, so that it is the same as the predetermined number of pixel categories.

[0073] Finally, the fused feature map is upsampled through the deconvolution layer to output the prediction map. However, upsampling only the last layer of feature maps will result in some loss of information in the original image. Therefore, it is necessary to Figure 4As shown in the figure, the intermediate feature maps of the convolutional network are upsampled multiple times (for example, the two intermediate feature maps of the 3×3 convolutional network are upsampled 4 times and 8 times respectively), and then the processed feature maps are integrated into the final downsampled feature map, so that the final generated probability prediction map can include both global information and rich detail information.

[0074] Based on the above steps S101 and S102, by constructing a lightweight mACNN model structure integrating dilated convolution, the mACNN model integrating the "shortcut" mechanism can effectively prevent the occurrence of gradient vanishing phenomenon while reducing the number of model parameters, and can also increase the receptive field of the model, thereby improving the segmentation performance of the model.

[0075] Compared with models such as Transformer and Bert, the requirements for computing resources of the constructed network model are greatly reduced. The proposed convolutional model uses dilated convolution, which can significantly increase the receptive field, help capture features of different scales, and make the segmentation results more accurate. At the same time, since dilated convolution does not increase the number of parameters, it can also reduce the computational burden of the model to a certain extent.

[0076] In step S103, based on the probability prediction graph, a multi-level conditional random field potential energy function is generated by constructing a potential energy function of the multi-level conditional random field model.

[0077] In some optional embodiments of this example, in order to improve the performance of the system, we use a multilevel conditional random field model (MLCRF) based on superpixel level and pixel level to optimize the mACNN model. The MLCRF model can make full use of spatial context information at different granularities, thereby enhancing the segmentation ability of the model.

[0078] Specifically, in order to more accurately model the high-dimensional complex features in remote sensing images, the MLCRF model introduces a high-order potential term based on the robust Potts model. Different from the classic Potts model, the potential function of the robust Potts model introduces a linear truncation function, allowing pixels with inconsistent labels to retain high-dimensional complex feature label values ​​different from the dominant label, thereby improving the robustness of the model.

[0079] Afterwards, the Canny edge detection model is used to capture the target contour features in the original image, and the edge potential energy function is constructed.

[0080] At the same time, the Simple Linear Iterative Clustering (SLIC) algorithm is used to perform superpixel segmentation on the original image to generate a superpixel segmentation map, and it is assumed that the label of each superpixel is the dominant category label within the superpixel.

[0081] Next, according to the superpixel segmentation map, the pixel-level probability prediction map output by the mACNN model is converted into a superpixel-level probability prediction map.

[0082] In some optional embodiments of this example, a superpixel unary potential energy function of the MLCRF model may be further defined based on the superpixel level probability prediction map.

[0083] The construction of superpixel binary potential function requires extracting superpixel level color features, spatial features and texture features from the superpixel segmentation map.

[0084] Although the superpixel-based conditional random field model can improve the segmentation speed, it usually requires that all pixels within a superpixel have the same classification label, which makes the segmentation result highly dependent on the quality of the clustering algorithm. For example, if the granularity of superpixel segmentation is not set properly, a superpixel may contain multiple objects at the same time, resulting in a decrease in the accuracy of the final segmentation.

[0085] To solve this problem, we added a pixel-level potential energy function constraint term to the superpixel-based Gaussian potential energy function to form a combined function that can fully utilize spatial context information at different granularities and avoid the negative impact of unsatisfactory superpixel segmentation results on the final segmentation performance.

[0086] Specifically, based on the high-dimensional complex feature label values, the pixel-level probability prediction map output by the mACNN model is used to construct a pixel-level potential function. However, since only the feature information of a single independent pixel is considered in the pixel-level potential function, the image segmentation results often contain noise and are not smooth. To solve this problem, the MLCRF model also extracts pixel-level color and spatial features from the original image and further constructs a pixel-level binary potential function.

[0087] In this way, the superpixel unary potential energy function, the superpixel binary potential energy function, the pixel unary potential energy function, and the pixel binary potential energy function together constitute the multi-level conditional random field potential energy function of the MLCRF model.

[0088] In step S104, based on the image segmentation performance, the constraints of the multi-level conditional random field potential function are set to generate a combined potential function, and the extreme value of the combined potential function is solved to complete the semantic segmentation prediction of the original image.

[0089] In order to solve the problem of improper setting of superpixel segmentation granularity, that is, a superpixel may contain multiple targets, resulting in a decrease in segmentation accuracy, the edge potential energy function is firstly integrated into the pixel unary potential energy function and pixel binary potential energy function of the MLCRF model, so as to perform edge constraints on the latter and generate a pixel multivariate potential energy function.

[0090] Next, we apply the pixel multivariate potential function to the superpixel unary potential function and the superpixel binary potential function, constraining their segmentation directions, and obtain a combined potential function that can fully utilize spatial context information at different granularities, thereby improving the segmentation effect.

[0091] Finally, the semantic segmentation of the original image is completed by solving the extreme value of the combined potential energy function in the MLCRF model.

[0092] Thus, based on step S103 and step S104, the embodiment of the present invention constructs an MLCRF model fused with a Canny edge feature extraction model as a backend optimization module. The model can make full use of spatial context information at different granularities to optimize the edge refinement and precision processing of the preliminary prediction map. It considers the connection between each pixel (superpixel) and the remaining pixels (superpixels), and constructs a mutual dependence relationship on all pixel pairs (superpixel pairs) in the image, thereby further extracting the global context information of the preliminary prediction map.

[0093] In addition, the present invention uses the Canny edge feature extraction model to capture the contour feature information of the target object, and integrates the boundary details into the edge potential function of the multi-level conditional random field model, thereby obtaining a back-end optimization model based on MLCRF. This allows the framework proposed in the embodiment of the present invention to fully utilize the powerful feature extraction capability of the convolutional neural network model while also utilizing the efficient learning and reasoning capabilities of the conditional random field model, thereby reducing the amount of calculation while improving the model prediction accuracy.

[0094] In a specific embodiment of the present invention, the training and testing process of the combined model for semantic segmentation based on a multi-scale dilated convolutional neural network and a multi-level conditional random field is also described in detail.

[0095] In the training phase, the dataset first needs to be divided into training set and test set.

[0096] The training set is used to train the model, and the test set is used to evaluate the performance and generalization ability of the model. In the first step, by using the training set to train the mACNN model, it is necessary to first clarify that the goal of the mACNN model is to generate a feature map of the same size as the original image. The number of channels of this feature map corresponds to the number of categories in the classification problem, and each pixel value in each channel represents the probability that the pixel belongs to that category. Therefore, the mACNN model not only needs to extract features of different scales in the image, but also needs to learn how to predict the category of each pixel from these features.

[0097] In the actual training process, multi-scale atrous convolutional neural networks are used to extract multi-level features of images. By convolution processing images at different scales, mACNN can capture fine-grained local features and global information over a larger range. This process can be trained through the back-propagation algorithm, which continuously optimizes the parameters of the model to make the predicted category probability map more and more accurate. Among them, the back-propagation algorithm calculates the gradient of the loss function and gradually adjusts the network parameters to minimize the prediction error during the training process.

[0098] After completing the training of the mACNN model, the next task is to train an MLCRF model that can integrate edge information. The core idea of ​​MLCRF is to optimize pixel-level label assignment through conditional random fields, especially when dealing with image edges and details. In order for MLCRF to work better, the probability prediction map generated by mACNN needs to be used as input. These preliminary probability maps are not perfect, and there may be some unclear edges or inconsistent category labels, so further optimization is needed.

[0099] In this process, the Canny edge detection algorithm plays an important role. The Canny algorithm can accurately detect the edge information in the image and provide MLCRF with rich edge features. These edge information are particularly important for subsequent segmentation optimization, because in semantic segmentation, the edges of objects are usually very critical areas, especially when dealing with complex objects or boundaries between adjacent categories.

[0100] In addition to edge information, the SLIC clustering algorithm also plays a key role in the training of MLCRF. Through this superpixel segmentation method, the image is divided into multiple uniformly sized and regularly shaped regions, which can usually better preserve the local features of the image. By incorporating these superpixel regions into the potential energy function of the conditional random field together with the edge information, MLCRF is able to find a balance between global and local information, thereby classifying pixels more accurately.

[0101] Specifically, the MLCRF model adjusts the relationship between pixel labels by constructing multiple potential functions. Each potential function contains information at different levels, such as edge information, characteristics of superpixel regions, etc. These potential functions are weighted to form the final combined potential function, and then the graph cut algorithm is used to solve the extreme value of the function to obtain the optimized segmentation result. Among them, the role of the graph cut algorithm here is to convert the potential function into a minimum cut problem of the graph, and by performing the minimum cut operation on the graph, the optimal configuration of label consistency between pixels is obtained.

[0102] After the training phase is completed, the next task is the testing phase of the model. In the testing phase, the images in the test set are first input into the trained mACNN model. The mACNN model processes the test images according to the multi-scale feature extraction method learned during the training process and generates a category probability prediction map for each pixel. This prediction map represents the model's estimate of the probability that each pixel in the image belongs to a different category.

[0103] Then, the generated probability prediction graph will be passed to the trained MLCRF model for further optimization. At this point, we also need to process the original image through the Canny edge detection algorithm and the SLIC clustering algorithm to obtain the corresponding edge information and superpixel division results. This information will be combined with the probability prediction Figure 1 The image is then input into the MLCRF model. MLCRF optimizes the classification results by adjusting the label consistency between each pixel and using edge and superpixel information. The combined potential energy function is solved by the graph cut algorithm, and the optimized segmentation result is finally obtained.

[0104] The core of the entire testing process lies in how to effectively combine deep learning models and traditional image processing methods. The mACNN model extracts multi-scale features of the image through deep learning and provides a preliminary probability prediction map for the MLCRF model. The MLCRF model further improves the accuracy of segmentation through the optimization process of conditional random fields, especially in the recognition of edges and detail areas. Ultimately, after this series of processing and optimization, the segmentation results obtained have higher accuracy and robustness, and can better cope with complex image segmentation tasks.

[0105] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0106] Reference Figure 5 , shows a structural block diagram of a semantic segmentation device 500 based on a multi-scale hole convolutional neural network and a multi-level conditional random field of the present application, the device comprising:

[0107] A multi-scale atrous convolutional neural network model construction module 501, used to construct a network layer of a multi-scale convolutional neural network;

[0108] An image feature extraction module 502 is used to extract features from the original image;

[0109] A potential energy function construction module 503 is used to construct a potential energy function of a multi-level conditional random field model;

[0110] The semantic segmentation module 504 is used to set the constraint conditions of the multi-level conditional random field potential energy function, generate a combined potential energy function, and solve the extreme value of the combined potential energy function.

[0111] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0112] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0113] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0114] Figure 6 800 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0115] Reference Figure 6, the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0116] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0117] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0118] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0119] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0120] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0121] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0122] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0123] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0124] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0125] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0126] Figure 7 19 is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0127] Reference Figure 7 , the computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0128] The computer readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer readable storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0129] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0131] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0134] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0137] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0138] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A semantic segmentation method based on multi-scale dilated convolutional neural network and multi-level conditional random field, characterized in that: The method comprises: Based on the residual network, a multi-scale atrous convolutional neural network model is generated by constructing the network layer of the multi-scale convolutional neural network. Based on the multi-scale hole convolutional neural network model, a probability prediction graph is generated by extracting features from the original image; Based on the probability prediction graph, a multi-level conditional random field potential energy function is generated by constructing a potential energy function of the multi-level conditional random field model; Based on the image segmentation performance, the constraints of the multi-level conditional random field potential energy function are set to generate a combined potential energy function, and the extreme value of the combined potential energy function is solved to complete the semantic segmentation prediction of the original image.

2. The method according to claim 1, characterized in that The multi-scale atrous convolutional neural network model includes an input layer, a atrous convolutional layer, a pooling layer, and a deconvolutional layer, wherein: Based on the residual network, the number of convolution kernel layers is skipped by a progressive mechanism to generate a hole convolution layer; Based on the spatial structure information of the image, the deconvolution layer is generated by replacing the fully connected layer.

3. The method according to claim 2, characterized in that By extracting features from the original image, including: Based on the nearest neighbor interpolation method, the original image is resized through the input layer to generate a standard image; Based on a preset convolution kernel size, the standard image is captured using the dilated convolution layer to generate feature structure information; Based on the feature structure information, a fused feature map is generated by adaptively adjusting the layer parameters of the pooling.

4. The method according to claim 3, characterized in that By extracting features from the original image, including: Based on the number of predetermined pixel categories, the number of feature channels is generated by performing a convolution operation on the number of channels; Based on the number of feature channels, the intermediate image of the fused feature map is multiplied by upsampling through the deconvolution layer to generate a probability prediction map.

5. The method according to claim 1, characterized in that By constructing the potential energy function of the multi-level conditional random field model, including: Based on the edge detection algorithm, the edge potential energy function is generated by extracting the contour feature information of the original image; Based on a simple linear iterative clustering algorithm, a superpixel segmentation map is generated by performing superpixel segmentation on the original image; Based on the superpixel segmentation map, generating a superpixel probability map by converting the probability prediction map; Based on the robust Potts model, high-dimensional complex feature label values ​​are generated by adding linear truncation function labels to the high-dimensional complex features in the image.

6. The method according to claim 5, characterized in that By constructing the potential energy function of the multi-level conditional random field model, including: Based on the superpixel probability map, a superpixel unary potential energy function is generated by defining a superpixel level unary potential energy function of a multi-level conditional random field model; Based on the high-dimensional complex feature label value, a superpixel binary potential function is generated by extracting superpixel level features of the superpixel segmentation map; Based on the high-dimensional complex feature label value, a pixel unary potential energy function is generated by modeling the pixel-level features of the probability prediction map; Based on the high-dimensional complex feature label value, by extracting the pixel-level features of the original image, a pixel binary potential energy function is generated; By integrating the superpixel unary potential energy function, the superpixel binary potential energy function, the pixel unary potential energy function, and the pixel binary potential energy function, a multi-level conditional random field potential energy function is generated.

7. The method according to claim 6, characterized in that By setting the constraint conditions of the multi-level conditional random field potential energy function, generating a combined potential energy function, and solving the extreme value of the combined potential energy function, including: Based on the edge potential energy function, generating a pixel multivariate potential energy function by performing edge constraints on the pixel unary potential energy function and the pixel binary potential energy function; Based on the pixel multi-element potential energy function, generating a combined potential energy function by performing segmentation constraints on the super-pixel unary potential energy function and the super-pixel binary potential energy function; Based on the image segmentation performance, the semantic segmentation prediction of the original image is completed by solving the extreme points of the combined potential energy function.

8. A semantic segmentation device based on a multi-scale dilated convolutional neural network and a multi-level conditional random field, characterized in that: include: Multi-scale hole convolutional neural network model construction module, used to construct network layers of multi-scale convolutional neural networks; Image feature extraction module, used to extract features from original images; Potential energy function construction module, used to construct the potential energy function of the multi-level conditional random field model; The semantic segmentation module is used to set the constraint conditions of the multi-level conditional random field potential energy function, generate a combined potential energy function, and solve the extreme value of the combined potential energy function.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images