A real-time semantic segmentation method and system based on multi-resolution branches

Abstract

The application discloses a kind of real-time semantic segmentation method and system based on multi-resolution branch, it is related to graphics processing technical field.By obtaining image data, using depth multi-resolution network to carry out semantic segmentation to image data, obtain semantic segmentation feature map;Wherein, semantic segmentation process includes: to image data is carried out convolution feature extraction, obtains the spatial information and semantic information of image;Using different resolution branch network to the spatial information and semantic information of image respectively learns to obtain the feature map of each branch network;Fusion is the feature map of each branch network, obtains the final semantic segmentation feature map.The application extracts the semantic and spatial information of image from different resolution respectively, solves the problem of insufficient utilization of semantic and spatial information.The application also adds semantic enhancement module and feature fusion module, improve the effect of semantic information extraction while also solve the problem of spatial information being covered under multi-branch.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a real-time semantic segmentation method and system based on multi-resolution branches. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] As the core field of artificial intelligence research, the development of computer vision has contributed greatly to the development of artificial intelligence. Computer vision is composed of image classification, detection, segmentation and other tasks, and is a field with wide coverage and multiple content. Semantic segmentation, as a cornerstone technology of computer vision, starts from a single pixel point and explores the semantic information contained between each pixel in the image, and identifies the objects in the image through semantic information. Semantic segmentation plays an important role in applications such as autonomous driving, remote sensing satellite image processing, and precision agriculture.

[0004] Semantic segmentation is to label each pixel point in the target class of the image according to "semantic", so that different classes are separated on the picture, which can also be understood as a pixel-level image classification task. The network used for semantic segmentation is mainly a convolutional neural network. Convolutional networks have the weakness of being difficult to capture long-distance dependencies, and solving the problem of long-distance dependencies can make the network obtain better performance. Solving this problem often requires stacking more convolutional layers and using larger receptive fields, but this will also lead to an increase in the size, parameter quantity and complexity of the network model and a decrease in the inference speed of the network model. In downstream tasks such as autonomous driving, real-time performance is a key requirement. The size and inference speed of the network model should be considered together with the performance and accuracy of the network model, and even should be given priority. And the method of improving the detection accuracy of the semantic segmentation task often uses a larger network architecture, such as ResNet network, which improves the accuracy but also brings the problem of mobile devices being difficult to deploy due to limited conditions. Therefore, how to realize a smaller model, fewer parameters, faster speed while maintaining good performance in the semantic segmentation task has become a problem to be solved in the prior art.

[0005] The inventors have found that the current network architecture has the following technical problems in the use of information of different resolutions, the extraction of semantic information and the fusion of feature maps of different branches:

[0006] (1) The more semantic and spatial information extracted by the model, the more knowledge the model itself can learn, which helps to improve the accuracy. But the existing network architecture does not make full use of the semantic and spatial information extracted by different branches, which limits the learning of the model and affects the accuracy of the semantic segmentation task.

[0007] (2) In semantic information extraction, the existing network structure designs some enhanced modules, but the performance of these enhanced modules cannot guarantee the consideration of model accuracy and model detection speed.

[0008] (3) In the fusion of feature maps of different branches, the existing model mostly uses the direct addition of feature maps, which will cover the spatial information of the semantic information extracted by the model, resulting in fuzzy edge contour of different objects in segmentation. SUMMARY

[0009] In view of the deficiencies of the prior art, the purpose of the present application is to provide a real-time semantic segmentation method and system based on multi-resolution branches, and a new deep multi-resolution network (DMRNet) is proposed, which increases more branches and fully utilizes the semantic and spatial information under different resolutions. The present application also adds a semantic enhancement module and a feature fusion module, which improves the effect of semantic information extraction and solves the problem of spatial information being covered under multiple branches.

[0010] In order to achieve the above purpose, the present application is realized by the following technical scheme:

[0011] The first aspect of the present application provides a real-time semantic segmentation method based on multi-resolution branches, comprising the following steps:

[0012] Obtain image data, and use a deep multi-resolution network to perform semantic segmentation on the image data to obtain a semantic segmentation feature map;

[0013] The process of using a deep multi-resolution network to perform semantic segmentation on the image data comprises:

[0014] Perform convolution feature extraction on the image data to obtain spatial information and semantic information of the image;

[0015] Use branch networks of different resolutions to learn the spatial information and semantic information of the image respectively to obtain feature maps of the branch networks;

[0016] Fuse the feature maps of the branch networks to obtain a final semantic segmentation feature map.

[0017] Further, the branch network comprises four layers of A, B, C and D branches, and the image resolutions in the four layers of A, B, C and D branch networks decrease in turn.

[0018] Further, the image resolution of the A-layer branch network is 1 / 8, the image resolution of the B-layer branch network is 1 / 16, the image resolution of the C-layer branch network is 1 / 32, and the image resolution of the D-layer branch network is 1 / 64.

[0019] Further, the specific steps of learning the spatial information and the semantic information of the image by the branch networks with different resolutions to obtain the feature maps of the branch networks are as follows:

[0020] According to the information extraction principle that the spatial information gradually decreases and the semantic information gradually increases, the A-D layer branch networks are used to learn the spatial information and the semantic information of the image with different resolutions in sequence;

[0021] The A-D layer branch networks output the image high-resolution spatial information feature map, the image medium-high resolution spatial information feature map, the image medium-low resolution semantic information feature map, and the image low-resolution semantic information feature map, respectively.

[0022] Further, the boundary loss calculation is performed on the image spatial information learned by the A-layer branch network.

[0023] Further, the feature map learned by the D-layer branch network is enhanced to obtain the semantic information feature map, and the enhancement process is as follows:

[0024] The feature map learned by the D-layer branch network is projected and convolved to increase the dimension;

[0025] The feature information is extracted from the feature map after the dimension increase through deep convolution;

[0026] The extracted feature information is projected and convolved to decrease the dimension;

[0027] The feature information after the dimension decrease and the original input feature map information are integrated to obtain the semantic information feature map.

[0028] Further, the specific steps of fusing the feature maps of the branch networks to obtain the final semantic segmentation feature map are as follows:

[0029] The feature map obtained by the A-layer branch network is processed to obtain an A-layer probability matrix;

[0030] The A-layer probability matrix is multiplied by the feature map obtained by the B-layer branch network to obtain a B-layer probability feature map;

[0031] The B-layer probability feature map is processed to obtain a B-layer probability matrix;

[0032] The B-layer probability matrix is multiplied by the feature map obtained by the C-layer branch network and the feature map obtained by the D-layer branch network, respectively, to obtain a C-layer probability feature map and a D-layer probability feature map;

[0033] The C-layer probability feature map and the D-layer probability feature map are added to the B-layer probability feature map to obtain a final semantic segmentation feature map.

[0034] The second aspect of the present application provides a real-time semantic segmentation system based on multi-resolution branches, comprising:

[0035] The semantic segmentation module is configured to acquire image data, perform semantic segmentation on the image data by using a deep multi-resolution network, and obtain a semantic segmentation feature map;

[0036] The deep multi-resolution network comprises:

[0037] The feature extraction module is configured to perform convolutional feature extraction on the image data to obtain spatial information and semantic information of the image;

[0038] The branch network module is configured to learn the spatial information and the semantic information of the image by using branch networks of different resolutions to obtain feature maps of the branch networks;

[0039] The feature fusion module is configured to fuse the feature maps of the branch networks to obtain a final semantic segmentation feature map.

[0040] The third aspect of the present application provides a medium having a program stored thereon, wherein the program is executed by a processor to implement the steps in the real-time semantic segmentation system based on multi-resolution branches according to the first aspect of the present application.

[0041] The fourth aspect of the present application provides a device comprising a memory, a processor, and a program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps in the real-time semantic segmentation system based on multi-resolution branches according to the first aspect of the present application.

[0042] The above one or more technical solutions have the following beneficial effects:

[0043] The present application discloses a real-time semantic segmentation method and system based on multi-resolution branches, which designs a model into multiple branches to extract semantic and spatial information of an image from different resolutions, thereby solving the problem of insufficient utilization of semantic and spatial information.

[0044] The present application designs a new semantic information enhancement module to enhance the output semantic information, improves the model precision, and also considers the detection speed of the model, thereby solving the performance problem of the previous enhancement module.

[0045] The present application fuses the features of different branch networks by using the feature fusion module, so that the information fusion effect of different branches is more effective, and the problem that the spatial information of the feature map is covered by the semantic information is solved.

[0046] Advantages of the additional aspects of the application will become apparent in the following description, which is given by way of example only, from the description, from the accompanying drawings and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated by reference herein. The embodiments illustrated in the drawings are presented by way of example in the description.

[0048] Figure 1 A schematic diagram of a deep multi-resolution network structure in the embodiment of the application is shown in Figure 1.

[0049] Figure 2 A schematic diagram of a feature map enhancement process in the embodiment of the application is shown in Figure 2.

[0050] Figure 3 A schematic diagram of a feature map fusion process in the embodiment of the application is shown in Figure 3. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed description is merely exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0052] It is to be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0053] Embodiment I:

[0054] The embodiment I of the application provides a real-time semantic segmentation method based on multi-resolution branches, comprising the following steps:

[0055] Image data is acquired, and a deep multi-resolution network DMRNet is used to perform semantic segmentation on the image data to obtain a semantic segmentation feature map. The deep multi-resolution network is designed through a multi-branch path, and a semantic enhancement module and a feature fusion module are added. The detection speed is improved, and the parameter amount is greatly reduced compared with a large semantic segmentation network, while the detection accuracy is not inferior to that of the large network.

[0056] In the process of semantic segmentation detection, the multi-branch information extraction method of the application enables the deep multi-resolution network to consider the extraction of semantic information and spatial information from multiple different resolutions. In the semantic information extraction branch, the detection speed of the network is improved on the premise that the accuracy remains unchanged by improving the existing semantic enhancement module. In the feature map fusion stage of different branches, the accuracy of semantic segmentation is improved by designing a feature map fusion module.

[0057] The specific deep multi-resolution network for image data semantic segmentation process includes the following steps:

[0058] Step 1: Convolutional feature extraction is performed on the image data to obtain spatial information and semantic information of the image.

[0059] Step 2: Different resolution branch networks are used to learn the spatial information and semantic information of the image to obtain feature maps of each branch network.

[0060] Step 3: The feature maps of each branch network are fused to obtain the final semantic segmentation feature map.

[0061] In step 1, the network depth of deep learning is too shallow and the performance is not good, and too deep does not meet the real-time requirement. Therefore, the deep multi-resolution network is designed as L1-L5 five layers in this embodiment, as shown in Figure 1 The unit module of L1-L4 network is composed of two BasicBlock, and the unit module of the last L5 layer is composed of an inverted residual block. The arrow information between different blocks represents the channel number and resolution of the output feature map of the network block.

[0062] BasicBlock is a block in ResNet, which is composed of two 3x3 convolutions, and a shortcut is added after the second convolution to add the feature maps. The L0 layer is two 3x3 convolutions with a step of 2, which reduces the resolution of the original picture to 1 / 4. In the early stage of the network (L0L1 period), the extracted features of the network are not perfect, and early reduction of resolution will cause loss of important information, so this embodiment designs to reduce the resolution after L0, L1 and L2, when the extracted features of the network are reliable.

[0063] After the features are extracted by the convolution of the previous L0 and L1 blocks, the network starts to have different branches from the L2 layer. Then, the outputs of different branch networks are fused to obtain the final semantic segmentation result. In deep learning, the more the number of channels is, the better the learning ability of the network is. The structure of the inverted residual is to first increase the number of channels by 1x1 convolution and input them into 3x3 convolution to extract feature map information, and then reduce the number of channels by 1x1 convolution. This makes the 3x3 convolution use several times more channels to extract image information, which naturally increases the performance of the network.

[0064] In step 2, the image information extracted at different resolutions is different, and adding different information is beneficial to improve the detection performance. Currently, some networks only use 1 / 8 (high resolution) and 1 / 64 (low resolution) feature maps for addition, which ignores the information carried by the intermediate resolution. Therefore, the present application adds 1 / 16 and 1 / 32 resolution information to obtain more information and add them, so as to solve the problem of insufficient use of semantic and spatial information extracted by the existing network architecture.

[0065] Based on this, the embodiment adopts branch networks of different resolutions to learn the spatial information and semantic information of the image respectively to obtain the feature maps of each branch network. The branch network includes A, B, C and D four layers, and the image resolutions in the A, B, C and D four layer branch networks decrease in turn. The A layer branch maintains the image resolution at 1 / 8, the B layer branch maintains the image resolution at 1 / 16 after the down-sampling operation of the A layer, the C layer branch maintains the image resolution at 1 / 32 after the down-sampling operation of the B layer, and the D layer branch maintains the image resolution at 1 / 64 after the down-sampling operation of the C layer. The A layer includes three unit modules L3A, L4A and L5A, the B layer includes three unit modules L3B, L4B and L5B, the C layer includes two unit modules L4C and L5C, and the D layer includes one unit module L5D. According to the information extraction principle of gradually reducing spatial information and gradually increasing semantic information, the A-D layer branch network is used to learn the spatial information and semantic information of the image at different resolutions in turn. Among them, the A layer branch contains the most image spatial information, and the image spatial information learned by the A layer branch network is calculated by boundary loss to assist the network to learn the spatial information of the image.

[0066] Specific steps of boundary loss calculation are as follows: the image is processed by the whole model to output the result, the result is respectively input into the main loss (CE Loss, i.e. cross entropy loss) and the boundary loss (Boundary Loss) to obtain the loss result, and then the loss results are added to obtain the final loss of the model.

[0067] The D-layer branch contains the most image semantic information, and the output feature map of the D layer is input into the semantic information enhancement module DUPPM to enhance the semantic information of the feature map. Finally, the output feature maps of the four branches with different resolutions are input into the multi-branch fusion module MBFM, and the output feature map is input into the final classification head to complete the image semantic segmentation task. The classification head is composed of a 3x3 convolution and a 1x1 convolution. The 3x3 convolution is responsible for information extraction of the result of the fusion module, and the 1x1 convolution maps the channel number of the feature map output by the 3x3 convolution to a specific classification number. For example, using the CityScapes dataset of the autonomous driving image, the data is divided into 19 categories (the number 10 represents a car, and the number 1 represents a pedestrian, etc.). Therefore, the channel number of the feature map output after the 1x1 convolution is 19, and the points in the output feature map matrix can be 10, 1, etc. to represent that the pixel point can be a pedestrian or a car.

[0068] In the branch network of the embodiment, the outputs of the A-D layer branch networks contain spatial and semantic information, and the process of extracting information from shallow resolution (1 / 8) to deep resolution (1 / 64) is a process of gradually reducing spatial details (information) and gradually increasing semantic information. The A-D layer branch networks output image high-resolution spatial information feature maps, image medium-high resolution spatial information feature maps, image medium-low resolution semantic information feature maps, and image low-resolution semantic information feature maps, respectively. In the embodiment, the spatial information accounts for 90% and the semantic information accounts for 10% in the image high-resolution spatial information feature map output by the A layer branch network; the spatial information accounts for 60% and the semantic information accounts for 40% in the image medium-high resolution spatial information feature map output by the B layer branch network; the spatial information accounts for 40% and the semantic information accounts for 60% in the image medium-low resolution semantic information feature map output by the C layer branch network; and the spatial information accounts for 10% and the semantic information accounts for 90% in the image low-resolution semantic information feature map output by the D layer branch network.

[0069] The feature map learned by the D layer branch network is enhanced to obtain a semantic information feature map. In the embodiment, to solve the performance problem of the enhancement module, the following is proposed Figure 2Deep Unit Pyramid Pooling Module (DUPPM). The DUPPM module borrows the depthwise separable convolution of the inverted residual module. The depthwise separable convolution is combined by two parts, Depthwise (DW) and Pointwise (PW). First, the feature map learned by the D branch network is projected and upgraded in dimension (PW); the upgraded feature map is extracted by depthwise convolution (DW); the extracted feature information is projected and downgraded in dimension (PW); the downgraded extracted feature information and the original input feature map information are integrated to obtain the semantic information feature map.

[0070] More specifically, the module inputs a feature map with a small number of channels, extracts 1 / 128, 1 / 256, 1 / 512 resolution and global feature map information through different pooling layers, then upsamples these feature maps and upgrades the feature maps in dimension through 1*1 convolution. The upgraded feature map is input into the depthwise convolution (DW) to extract the feature map information, and is downgraded in dimension again through 1*1 convolution. Then the obtained feature map information and the original input feature map information are respectively subjected to concat operation and add operation to complete the extraction of semantic information, and an enhanced feature map is obtained.

[0071] In step 3, as shown in Figure 3 the specific steps of fusing the feature maps of each branch network to obtain the final semantic segmentation feature map are as follows:

[0072] The 1 / 8 resolution feature map of the A layer branch network is obtained The A layer probability matrix a is obtained by the sigmoid function; each value of the matrix represents the probability of whether a pixel point is a boundary.

[0073] The A layer probability matrix a is multiplied by the 1 / 16 resolution feature map of the B layer branch network to obtain the B layer probability feature map; the B layer probability feature map is processed to obtain the B layer probability matrix 1-a; 1-a represents the probability that a pixel point is a non-boundary.

[0074] The B layer probability matrix 1-a is multiplied by the 1 / 32 resolution feature map of the C layer branch network and the 1 / 64 resolution feature map of the D layer branch network respectively to obtain the C layer probability feature map and the D layer probability feature map.

[0075] The C layer probability feature map and the D layer probability feature map are added to the B layer probability feature map and input into the 3*3 convolution f to obtain the final semantic segmentation feature map, and the formula is as follows:

[0076]

[0077]

[0078] By the above method, the embodiment forces the network to learn whether to trust the branch with more spatial information or the branch with more semantic information by using the probability matrix α. Generally, when α>0.5, the network will trust the branch with more spatial information, and vice versa.

[0079] Embodiment two:

[0080] The embodiment two of the present application provides a real-time semantic segmentation system based on multi-resolution branches, comprising:

[0081] A semantic segmentation module is configured to acquire image data, perform semantic segmentation on the image data by using a deep multi-resolution network, and obtain a semantic segmentation feature map.

[0082] The deep multi-resolution network comprises:

[0083] A feature extraction module is configured to perform convolutional feature extraction on the image data to obtain spatial information and semantic information of the image.

[0084] A branch network module is configured to learn the spatial information and the semantic information of the image by using branch networks with different resolutions to obtain feature maps of the branch networks.

[0085] A feature fusion module is configured to fuse the feature maps of the branch networks to obtain a final semantic segmentation feature map.

[0086] Embodiment three:

[0087] The embodiment three of the present application provides a medium having a program stored thereon, and the program is executed by a processor to implement the steps in the real-time semantic segmentation method based on multi-resolution branches according to the embodiment one of the present application, and the steps are as follows:

[0088] Acquiring image data, performing semantic segmentation on the image data by using a deep multi-resolution network, and obtaining a semantic segmentation feature map.

[0089] The process of performing semantic segmentation on the image data by using the deep multi-resolution network comprises:

[0090] Performing convolutional feature extraction on the image data to obtain spatial information and semantic information of the image.

[0091] Learning the spatial information and the semantic information of the image by using branch networks with different resolutions to obtain feature maps of the branch networks.

[0092] fuse the feature maps of each branch network to obtain a final semantic segmentation feature map.

[0093] The detailed steps are the same as those of the real-time semantic segmentation method based on the multi-resolution branch provided in Embodiment 1, and thus are not described herein.

[0094] Embodiment 4

[0095] Embodiment 4 of the present application provides a device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps in the real-time semantic segmentation method based on the multi-resolution branch as described in Embodiment 1 of the present application when executing the program, and the steps are as follows:

[0096] obtaining image data, performing semantic segmentation on the image data by using a deep multi-resolution network to obtain a semantic segmentation feature map;

[0097] The process of performing semantic segmentation on the image data by using the deep multi-resolution network includes:

[0098] performing convolution feature extraction on the image data to obtain spatial information and semantic information of the image;

[0099] performing learning on the spatial information and the semantic information of the image by using branch networks of different resolutions to obtain feature maps of each branch network;

[0100] fusing the feature maps of each branch network to obtain a final semantic segmentation feature map.

[0101] The detailed steps are the same as those of the real-time semantic segmentation method based on the multi-resolution branch provided in Embodiment 1, and thus are not described herein.

[0102] The term “computer readable storage medium” should be understood to include a single medium or multiple media, that is, one or more sets of instructions; and should also be understood to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by a processor and that causes a processor to perform any one of the methods of the present application.

[0103] Those skilled in the art should understand that each module or step of the present application described above can be realized by a general computer device, and alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively manufactured into individual integrated circuit modules, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0104] The above describes the specific embodiments of the present application in combination with the drawings, but is not a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A real-time semantic segmentation method based on multi-resolution branches, characterized in that, The method comprises the following steps: obtaining image data, performing semantic segmentation on the image data by using a deep multi-resolution network to obtain a semantic segmentation feature map; wherein the process of performing semantic segmentation on the image data by using the deep multi-resolution network comprises: performing convolution feature extraction on the image data to obtain spatial information and semantic information of the image; learning the spatial information and the semantic information of the image by using branch networks of different resolutions to obtain feature maps of the branch networks; the branch networks comprise four layers of A, B, C and D branches, and the image resolutions in the four layers of A, B, C and D branch networks decrease in turn; performing boundary loss calculation on the spatial information of the image learned by the A-layer branch network to assist the network in learning the spatial information of the image; and inputting the output feature map of the D layer into a semantic information enhancement module to enhance the semantic information of the feature map; fusing the feature maps of the branch networks to obtain a final semantic segmentation feature map; the specific steps of fusing the feature maps of the branch networks to obtain the final semantic segmentation feature map are as follows: processing the feature map obtained by the A-layer branch network to obtain an A-layer probability matrix; multiplying the A-layer probability matrix with the feature map obtained by the B-layer branch network to obtain a B-layer probability feature map; processing the B-layer probability feature map to obtain a B-layer probability matrix; multiplying the B-layer probability matrix with the feature map obtained by the C-layer branch network and the feature map obtained by the D-layer branch network respectively to obtain a C-layer probability feature map and a D-layer probability feature map; adding the C-layer probability feature map and the D-layer probability feature map to the B-layer probability feature map to obtain the final semantic segmentation feature map.

2. The real-time semantic segmentation method based on multi-resolution branch according to claim 1, characterized in that, The image resolution of the A-layer branch network is 1 / 8, the image resolution of the B-layer branch network is 1 / 16, the image resolution of the C-layer branch network is 1 / 32, and the image resolution of the D-layer branch network is 1 / 64.

3. The real-time semantic segmentation method based on multi-resolution branch of claim 1, wherein, The specific steps of learning the spatial information and the semantic information of the image by using branch networks of different resolutions to obtain feature maps of the branch networks are as follows: according to the information extraction principle that the spatial information gradually decreases and the semantic information gradually increases, learning the spatial information and the semantic information of the image by using the A-D layer branch networks in different resolutions in turn; the A-D layer branch networks output image high-resolution spatial information feature maps, image medium-high resolution spatial information feature maps, image medium-low resolution semantic information feature maps and image low-resolution semantic information feature maps respectively.

4. The real-time semantic segmentation method based on multi-resolution branch of claim 1, wherein, enhancing the feature map learned by the D-layer branch network to obtain a semantic information feature map, and the enhancement process comprises: performing projection convolution dimension increasing on the feature map learned by the D-layer branch network; extracting feature information from the dimension-increased feature map by deep convolution; performing projection convolution dimension reduction on the extracted feature information; integrating the dimension-reduced extracted feature information and the original input feature map information to obtain a semantic information feature map.

5. A real-time semantic segmentation system based on multi-resolution branches, characterized in that, comprise: a semantic segmentation module configured to obtain image data, perform semantic segmentation on the image data by using a deep multi-resolution network, and obtain a semantic segmentation feature map; wherein the deep multi-resolution network comprises: a feature extraction module configured to perform convolution feature extraction on the image data to obtain spatial information and semantic information of the image; The branch network module is configured to learn spatial information and semantic information of an image respectively by branch networks with different resolutions to obtain feature maps of the branch networks; the branch network includes four layers of A, B, C and D branches, and image resolutions in the four layers of A, B, C and D branch networks decrease successively; Boundary loss calculation is performed on the spatial information of the image learned by the A-layer branch network to assist the network in learning the spatial information of the image; and the output feature map of the D layer is input into a semantic information enhancement module to enhance the semantic information of the feature map; The feature fusion module is configured to fuse the feature maps of the branch networks to obtain a final semantic segmentation feature map; The specific steps of fusing the feature maps of the branch networks to obtain the final semantic segmentation feature map are as follows: The feature map obtained by the A-layer branch network is processed to obtain an A-layer probability matrix; The A-layer probability matrix is multiplied by the feature map obtained by the B-layer branch network to obtain a B-layer probability feature map; The B-layer probability feature map is processed to obtain a B-layer probability matrix; The B-layer probability matrix is multiplied by the feature map obtained by the C-layer branch network and the feature map obtained by the D-layer branch network respectively to obtain a C-layer probability feature map and a D-layer probability feature map; The C-layer probability feature map and the D-layer probability feature map are added to the B-layer probability feature map to obtain a final semantic segmentation feature map.

6. A computer-readable storage medium, characterized in that, A plurality of instructions are stored in the computer readable storage medium, and the instructions are adapted to be loaded and executed by the processor of the terminal device to implement the real-time semantic segmentation method based on the multi-resolution branch in any one of claims 1-4.

7. A terminal device, characterized by, The computer readable storage medium is used to store a plurality of instructions, and the instructions are adapted to be loaded and executed by the processor to implement the real-time semantic segmentation method based on the multi-resolution branch in any one of claims 1-4.

Images