A semantic segmentation method, system and device with a three-branch network structure

Abstract

The present invention provides a semantic segmentation method, system and device of a three-branch network structure, which relate to the field of visual perception technology, and include the following steps: extracting an original feature map and inputting it into a trained three-branch network structure model through preprocessing to obtain a semantic segmentation result of a robot visual image, processing image information through the three-branch network structure model, and using a detail branch to parse and save detail information of the original feature map; using a context branch to extract deep semantic information and aggregate long-range information; and using a boundary branch to extract information with large pixel changes and identify boundaries. The present invention can accurately identify object boundaries, alleviate the phenomenon that the same object is interrupted by other objects, and can segment and identify objects at a real-time speed. The present invention effectively meets the requirements of segmentation accuracy and prediction speed.

Description

Technical Field

[0001] The present invention relates to a semantic segmentation method, system and device of a three-branch network structure, belonging to the technical field of robot visual perception. Background Art

[0002] Semantic segmentation is an important part of the robot's visual perception system. Semantic segmentation is to assign a category ID to each pixel in the image according to the object it belongs to, and to group together the parts of the image that belong to the same target. This allows the robot to know the semantic information of the target. By combining semantic information with point cloud information, judging whether targets at different distances from the robot are obstacles is a prerequisite for the robot's planning, control, and obstacle avoidance decisions; real-time acquisition and quality assurance of semantic information are key to improving the efficiency and quality of robot decision-making.

[0003] Semantic segmentation methods with three-branch network structures are divided into traditional methods and deep learning methods. Traditional semantic segmentation methods with three-branch network structures, such as grayscale segmentation and conditional random fields, require manual feature extraction and optimization to correctly learn the representation of complex classes; deep learning methods can complete semantic segmentation tasks end-to-end without manual feature extraction, and their performance has surpassed traditional methods. At present, the efficiency of semantic segmentation methods with three-branch network structures is generally improved by lightweight encoders and decoders (convolution factorization, group convolution), multi-scale input and dual-branch networks. In terms of improving the quality of semantic segmentation, a large receptive field is generally used to obtain context information, and spatial detail information is used to accurately identify the target boundary. Conventional operations such as encoder cascade convolution or pooling increase the receptive field and improve global dependence. The decoder upsamples to obtain semantic information, and through dilated convolution, the receptive field is increased and spatial detail features are extracted while maintaining spatial resolution. However, the resolution of spatial detail output and context output is very different. Direct fusion will cause overshoot, and the boundary information of the target will be interrupted by surrounding pixels, and small targets are easily affected by large targets. Summary of the invention

[0004] The purpose of the present invention is to provide a semantic segmentation method, system and device with a three-branch network structure, which balances context information and detail features so that the boundary semantics of adjacent targets are inconsistent, and the semantics of pixels in the same category are not interrupted by the semantics of pixels in other categories. It has higher accuracy and good real-time performance, and can meet the real-time and accurate segmentation target requirements.

[0005] In order to achieve the above objectives / solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0006] In a first aspect, the present invention provides a semantic segmentation method of a three-branch network structure, comprising:

[0007] Use the cascaded residual backbone network to extract features from robot vision images and obtain the corresponding original feature map;

[0008] Performing downsampling and semantic enrichment processing on the original feature map to obtain a processed feature map;

[0009] Inputting the processed feature graph into a trained three-branch network structure model, wherein the three-branch network structure model includes: a detail branch, a context branch, and a boundary branch;

[0010] The detail branch is used to parse the detail information of the processed feature map to obtain the detail feature map of the robot vision picture; the boundary branch is used to extract the pixel change information of the processed feature map, identify the boundary, and obtain the boundary feature map of the robot vision picture; the context branch is used to extract the deep semantic information of the processed feature map and aggregate the long-range information to obtain the context feature map of the robot vision picture, and the obtained detail feature map, boundary feature map and context feature map are balanced to obtain a balanced feature map;

[0011] The balanced feature map is then processed with fine semantic information to obtain the semantic segmentation result of the robot vision image.

[0012] Further, the process of using the context branch to extract the deep semantic information of the processed feature map and aggregate the long-range information to obtain the context feature map of the robot visual image includes: aggregating the feature map F9 after three times of downsampling and semantic enrichment of the processed feature map F1 to obtain the context feature map F11;

[0013] The method of using the detail branch to parse the detail information of the processed feature map to obtain the detail feature map of the robot visual image includes: subjecting the processed feature map F1 to fine-grained residual enriched semantic processing to obtain a feature map F2, and fusing the feature map F2 with the feature map obtained after the processed feature map F1 is processed once and twice by the context branch through downsampling enriched semantic processing to obtain a detail feature map F8;

[0014] The method of using the boundary branch to extract pixel change information of the processed feature map, identify boundaries, and obtain the boundary feature map of the robot vision image includes: subjecting the processed feature map F1 to coarse-grained residual enriched semantic processing to obtain a feature map F4, and sequentially adding the feature map F4 to the feature map obtained after the processed feature map F1 is processed once and twice by downsampling enriched semantic processing through the context branch to obtain a boundary feature map F10.

[0015] Further, the feature map F2 is sequentially fused with the feature map F1 obtained after one and two downsampling semantic enrichment processes through the context branch to obtain the detail feature map F8, which specifically includes the following steps:

[0016] The feature map F2 is fused with the feature map F3 obtained after the processed feature map F1 is processed by downsampling once through the context branch and semantic enrichment, and the feature map F5 is obtained. The corresponding expression is as follows:

[0017] σ=Sigmoid(f d (F2)f c (F3)),

[0018] F5=(1-σ)F2+σF3,

[0019] Where: σ is the output of the Sigmoid function, F5 is the feature map output after the fusion of feature map F2 and feature map F3;

[0020] f d is a combination of convolution, batch normalization, and ReLu operations, f d The convolution kernel size is 3 and the stride is 1;

[0021] f c is a combination of convolution, batch normalization, ReLu, and upsampling operations, f c The convolution kernel size is 3 and the stride is 1;

[0022] The feature map F5 is fused with the feature map F6 obtained after the processed feature map F1 is processed by downsampling twice through the context branch and enriching the semantics, to obtain the detail feature map F8. The corresponding expression is as follows;

[0023] σ=Sigmoid(f d (F5)f c (F6)),

[0024] F8=(1-σ)F5+σF6.

[0025] Furthermore, the feature map F9 processed by downsampling the feature map F1 three times to obtain the context feature map F11 is aggregated, including:

[0026] The feature map F9 is input into four pooling layers respectively, and four feature maps are output through 1*1 convolution and upsampling operations;

[0027] The sum of the four feature maps and the feature map F21 obtained by the 1*1 convolution of the feature map F9 is respectively subjected to a 3*3 convolution operation;

[0028] Concatenate the 4 values ​​obtained from the 3*3 convolution operation with the feature map F21;

[0029] Add the value after the connection operation to the feature map F21 to obtain the context feature map F11 output by the aggregation process;

[0030] Among them: the four pooling layers are average pooling with a stride of 2 and a window size of 5; average pooling with a stride of 4 and a window size of 9; average pooling with a stride of 8 and a window size of 19 and a global pooling layer.

[0031] Furthermore, the acquired detail feature map, boundary feature map and context feature map are balanced to obtain an expression of a balanced feature map as follows:

[0032] σ=Sigmoid(F10),

[0033] Out=f out ((1-σ)F11+σF8),

[0034] Where: σ is the output of the Sigmoid function, F8 is the detail feature map, F10 is the boundary feature map, F11 is the context feature map, and Out is the output balanced feature map; f out It is a combination of convolution, batch normalization and ReLu operations, with a kernel size of 3 and a stride of 1.

[0035] Furthermore, the training method of the three-branch network structure model includes the following steps:

[0036] The loss function L1 is set according to the boundary feature map, the loss function L2 is set according to the context feature map, and the loss function L3 is set by combining the context feature map and the boundary feature map;

[0037] The total loss function is constructed based on the loss function L1, loss function L2, and loss function L3:

[0038] Loss=λ1L1+λ2L2+λ3L3,

[0039] Where: L1 is the weighted binary cross entropy loss function set for the boundary branch, L2 is the cross entropy loss function set for the context branch, and L3 is the boundary-aware loss function; λ1, λ2, λ3 are the weight parameters of L1, L2, and L3 respectively;

[0040]

[0041] Where: t is the predefined threshold, b i is the output of the boundary header;

[0042] S i,c is the segmentation truth value of the i-th pixel of category c; is the prediction result of the i-th pixel of category c,

[0043] The constructed three-branch network structure model is iteratively trained using a robot vision image dataset until the total loss function converges or reaches an upper limit on the number of iterations.

[0044] Furthermore, the fine-grained residual-rich semantic processing includes, in sequence: convolution, batch normalization, ReLu, convolution, batch normalization, residual connection, ReLu, convolution, batch normalization, ReLu, convolution, batch normalization, residual connection processing steps; wherein the convolution kernel is 3 and the stride is 1;

[0045] The coarse-grained residual enriched semantic processing includes: convolution, batch normalization, ReLu, convolution, batch normalization, residual connection, ReLu processing steps in sequence, wherein the convolution kernel is 3 and the stride is 1;

[0046] The semantic information fine processing includes: batch normalization, ReLu, 3*3 convolution, batch normalization, ReLu and 1*1 convolution processing steps in sequence.

[0047] Further, downsampling and semantic enrichment processing is performed on the original feature map to obtain a processed feature map, including:

[0048] The original feature map is processed by sequentially performing convolution, batch normalization, ReLu, convolution, batch normalization, downsampling, residual connection, ReLu, convolution, batch normalization, ReLu, convolution, batch normalization, and residual connection processing steps to obtain a resolution size of the original feature map. Figure 1 / 8 processing feature map;

[0049] Among them, the convolution kernel is 3 and the stride is 1.

[0050] In a second aspect, the present invention provides a semantic segmentation system with a three-branch network structure, comprising:

[0051] An extraction module is used to extract features from robot vision images using a cascaded residual backbone network to obtain the corresponding original feature map;

[0052] A downsampling module is used to perform downsampling and semantic enrichment processing on the original feature map to obtain a processed feature map;

[0053] A processing module is used to input a processed feature map into a trained three-branch network structure model, wherein the three-branch network structure model includes: a detail branch, a context branch and a boundary branch; the detail branch is used to parse the detail information of the processed feature map to obtain a detail feature map of a robot vision picture; the boundary branch is used to extract pixel change information of the processed feature map, identify boundaries, and obtain a boundary feature map of the robot vision picture; the context branch is used to extract deep semantic information and aggregate long-range information of the processed feature map to obtain a context feature map of the robot vision picture, and the obtained detail feature map, boundary feature map and context feature map are subjected to balancing processing to obtain a balanced feature map; the balanced feature map is further subjected to fine processing of semantic information to obtain a semantic segmentation result of the robot vision picture.

[0054] In a third aspect, the present invention provides a semantic segmentation device with a three-branch network structure, comprising: a processor and a storage medium;

[0055] The storage medium is used to store instructions;

[0056] The processor is used to operate according to the instructions to execute the steps of the semantic segmentation method of the three-branch network structure.

[0057] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention designs a three-branch robot semantic segmentation network, including a detail branch, a context branch and a boundary branch; the detail branch parses and saves the detail information of the high-resolution feature map; the context branch extracts deep context information and aggregates long-range information; the boundary branch extracts information with large pixel changes and identifies boundaries; the present invention can balance the context information and detail features, improve the quality of semantic segmentation, and improve the efficiency of semantic segmentation.

[0058] The present invention can accurately identify the boundary of an object, alleviate the phenomenon that the same object is interrupted by other objects, and can segment and identify the object at a real-time speed; the present invention effectively meets the requirements of segmentation accuracy and prediction speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of the overall network structure of a semantic segmentation method with a three-branch network structure according to the present invention;

[0060] Figure 2 A processing schematic diagram of a semantic segmentation method of a three-branch network structure of the present invention;

[0061] Figure 3 It is a schematic diagram of the structure of the pixel attention fusion module of the detail branch of the present invention;

[0062] Figure 4A schematic diagram of the context aggregation module structure of the context branch of the present invention;

[0063] Figure 5 A schematic diagram of the context and detail balancing module structure of the context branch of the present invention;

[0064] Figure 6 is the original image;

[0065] Figure 7 A schematic diagram of an image slice where a road pixel is truncated by other pixels;

[0066] Figure 8 Image schematic to alleviate the problem of road pixels being truncated by other pixels. DETAILED DESCRIPTION

[0067] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0068] The term "and / or" is only a way to describe the association relationship of related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0069] Example 1

[0070] like Figure 1 to Figure 2 An embodiment shown in FIG. 1 is a semantic segmentation method of a three-branch network structure, comprising:

[0071] The cascaded residual backbone network is used to extract features from the robot vision image and obtain the corresponding original feature map A;

[0072] Performing downsampling and semantic enrichment processing on the original feature map A to obtain a processed feature map;

[0073] Inputting the processed feature graph into a trained three-branch network structure model, wherein the three-branch network structure model includes: a detail branch, a context branch, and a boundary branch;

[0074] The detail branch is used to parse the detail information of the processed feature map to obtain the detail feature map of the robot vision picture; the boundary branch is used to extract the pixel change information of the processed feature map, identify the boundary, and obtain the boundary feature map of the robot vision picture; the context branch is used to extract the deep semantic information of the processed feature map and aggregate the long-range information to obtain the context feature map of the robot vision picture, and the obtained detail feature map, boundary feature map and context feature map are balanced to obtain a balanced feature map;

[0075] The balanced feature map is then processed with fine semantic information to obtain the semantic segmentation result of the robot vision image.

[0076] This embodiment uses a robot vision image data set and a three-branch network structure model constructed by loss function training; the robot vision image data set is obtained and divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0077] The loss function L1 is set according to the boundary feature map, the loss function L2 is set according to the context feature map, and the loss function L3 is set by combining the context feature map and the boundary feature map;

[0078] The total loss function is constructed based on the loss function L1, loss function L2, and loss function L3:

[0079] Loss=λ1L1+λ2L2+λ3L3,

[0080] Where: L1 is the weighted binary cross entropy loss function set for the boundary branch, L2 is the cross entropy loss function set for the context branch, and L3 is the boundary-aware loss function; λ1, λ2, λ3 are the weight parameters of L1, L2, and L3 respectively;

[0081]

[0082] Where: t is the predefined threshold, b i is the output of the boundary header;

[0083] S i,c is the segmentation truth value of the i-th pixel of category c; is the prediction result of the i-th pixel of category c,

[0084] The constructed three-branch network structure model is iteratively trained using a robot vision image dataset until the total loss function converges or reaches an upper limit on the number of iterations.

[0085] After preprocessing, the extracted original feature map A is input into the detail branch, context branch and boundary branch of the three-branch network structure model for further processing, including:

[0086] The specific operation of the extracted original feature map preprocessing is: the extracted original feature map A is processed by downsampling and semantic enrichment to obtain a resolution size of the original feature map Figure 1 / 8 feature map F1, where: the downsampling semantic enrichment processing includes: convolution, batch normalization, ReLu, convolution, batch normalization, downsampling, residual connection, ReLu, convolution, batch normalization, ReLu, convolution, batch normalization, residual connection processing steps, where the convolution kernel is 3 and the stride is 1;

[0087] The original feature map A is processed by three sets of downsampling and semantic enrichment to obtain a feature map F1 with a resolution of 1 / 8 of the original feature; wherein: the downsampling and semantic enrichment processing includes: convolution, batch normalization, ReLu, convolution, batch normalization, downsampling, residual connection, ReLu activation function, convolution, batch normalization, ReLu, convolution, batch normalization, residual connection processing steps, wherein the convolution kernel is 3 and the stride is 1;

[0088] The feature map F1 is input into the detail branch, and the feature map F2 is obtained after fine-grained residual enriched semantic processing; the fine-grained residual enriched semantic processing includes: convolution, batch normalization, ReLu, convolution, batch normalization, residual connection, ReLu, convolution, batch normalization, ReLu, convolution, batch normalization, residual connection processing steps, wherein the convolution kernel is 3 and the stride is 1;

[0089] The feature map F1 is input into the context branch, and the feature map F3 is obtained after downsampling and semantic enrichment processing; wherein: the feature map F3 is processed by downsampling and semantic enrichment once to obtain the feature map F6, and the feature map F9 is obtained after downsampling and semantic enrichment processing twice;

[0090] The feature map F1 is input into the boundary branch, and the feature map F4 is obtained after coarse-grained residual enriched semantic processing; the coarse-grained residual enriched semantic processing includes: convolution, batch normalization, ReLu, convolution, batch normalization, residual connection, ReLu processing, where the convolution kernel is 3 and the stride is 1.

[0091] like Figure 3 As shown, after the feature map F2 and the upsampled feature map F3 are input into the first pixel attention fusion module for fusion processing, the feature map F5 is obtained; after the feature map F5 and the upsampled feature map F6 are input into the second pixel attention fusion module for fusion processing, the detail feature map F8 is obtained, which specifically includes the following steps:

[0092] The feature map F2 is fused with the feature map F3 obtained after the processed feature map F1 is processed by downsampling once through the context branch and semantic enrichment, and the feature map F5 is obtained. The corresponding expression is as follows:

[0093] σ=Sigmoid(f d (F2)f c (F3)),

[0094] F5=(1-σ)F2+σF3

[0095] Where: σ is the output of the Sigmoid function, F5 is the feature map output after the fusion of feature map F2 and feature map F3;

[0096] f d is a combination of convolution, batch normalization, and ReLu operations, f d The convolution kernel size is 3 and the stride is 1;

[0097] f c is a combination of convolution, batch normalization, ReLu, and upsampling operations, f c The convolution kernel size is 3 and the stride is 1;

[0098] The feature map F5 is fused with the feature map F6 obtained after the processed feature map F1 is processed by downsampling twice through the context branch and enriching the semantics, to obtain the detail feature map F8. The corresponding expression is as follows;

[0099] σ=Sigmoid(f d (F5)f c (F6)),

[0100] F8=(1-σ)F5+σF6.

[0101] If σ is very high, it means that the context branch is semantically rich, which is determined by F3 and F6.

[0102] After adding the feature map F4 and the upsampled feature map F3, a feature map F7 is obtained;

[0103] After adding the feature map F7 and the upsampled feature map F6, a boundary feature map F10 is obtained;

[0104] After the feature map F9 is input into the context fast aggregation module, an aggregation process is performed, and the context feature map F11 after the aggregation process is input into the context and detail balance module together with the detail feature map F8 and the boundary feature map F10 for balance processing to obtain a balanced feature map F12;

[0105] like Figure 4 As shown, the feature map F9 processed by downsampling the feature map F1 three times in sequence and enriching the semantics is aggregated to obtain the context feature map F11, including:

[0106] The feature map F9 is input into four pooling layers respectively, and four feature maps are output through 1*1 convolution and upsampling operations;

[0107] The sum of the four feature maps and the feature map F21 obtained by the 1*1 convolution of the feature map F9 is respectively subjected to a 3*3 convolution operation;

[0108] Concatenate the 4 values ​​obtained from the 3*3 convolution operation with the feature map F21;

[0109] Add the value after the connection operation to the feature map F21 to obtain the context feature map F11 output by the aggregation process;

[0110] Among them: the four pooling layers are average pooling with a stride of 2 and a window size of 5; average pooling with a stride of 4 and a window size of 9; average pooling with a stride of 8 and a window size of 19 and a global pooling layer.

[0111] like Figure 5 As shown, the acquired detail feature map, boundary feature map and context feature map are balanced, and the expression of the balanced feature map is as follows:

[0112] σ=Sigmoid(F10),

[0113] Out=f out ((1-σ)F11+σF8),

[0114] Where: σ is the output of the Sigmoid function, F8 is the detail feature map, F10 is the boundary feature map, F11 is the context feature map, and Out is the output balanced feature map; f out It is a combination of convolution, batch normalization and ReLu operations, with a kernel size of 3 and a stride of 1.

[0115] Embodiment 2:

[0116] This embodiment provides a semantic segmentation system with a three-branch network structure, including:

[0117] An extraction module is used to extract features from robot vision images using a cascaded residual backbone network to obtain the corresponding original feature map;

[0118] A downsampling module is used to perform downsampling and semantic enrichment processing on the original feature map to obtain a processed feature map;

[0119] A processing module is used to input a processed feature map into a trained three-branch network structure model, wherein the three-branch network structure model includes: a detail branch, a context branch and a boundary branch; the detail branch is used to parse the detail information of the processed feature map to obtain a detail feature map of a robot vision picture; the boundary branch is used to extract pixel change information of the processed feature map, identify boundaries, and obtain a boundary feature map of the robot vision picture; the context branch is used to extract deep semantic information and aggregate long-range information of the processed feature map to obtain a context feature map of the robot vision picture, and the obtained detail feature map, boundary feature map and context feature map are subjected to balancing processing to obtain a balanced feature map; the balanced feature map is further subjected to fine processing of semantic information to obtain a semantic segmentation result of the robot vision picture.

[0120] Embodiment 3:

[0121] A semantic segmentation device with a three-branch network structure, comprising: a processor and a storage medium;

[0122] The storage medium is used to store instructions;

[0123] The processor is used to operate according to the instructions to execute the steps of the semantic segmentation method of the three-branch network structure.

[0124] like Figures 6 to 8 As shown in the figure, the semantic segmentation method of the three-branch network structure can identify 9 categories: cars, bicycles, people, roads, buildings, obstacles, vegetation, pits and neglect; it achieves mIoU = 0.69 on the self-made dataset.

[0125] Pixel_Acc=0.9669,Meac_Acc=0.7131, the inference speed on 3090 is 31fps, and the parameter volume is 36.9M; it can be seen from the figure that this method can recognize the target semantics in real time. The semantics of the pixels inside each target are consistent, and only become inconsistent at the boundaries of adjacent targets, alleviating the phenomenon that pixels of the same category are interrupted by pixels of other categories, especially ground segmentation.

[0126] The present invention designs a three-branch robot semantic segmentation network, including a detail branch, a context branch and a boundary branch; the detail branch parses and saves the detail information of the high-resolution feature map; the context branch extracts deep context information and aggregates long-range information; the boundary branch extracts information with large pixel changes and identifies boundaries. The present invention can balance the context information and detail features, improve the quality of semantic segmentation, and improve the efficiency of semantic segmentation.

[0127] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A semantic segmentation method with a three-branch network structure, characterized in that: include: Use the cascaded residual backbone network to extract features from robot vision images and obtain the corresponding original feature map; Performing downsampling and semantic enrichment processing on the original feature map to obtain a processed feature map; Inputting the processed feature graph into a trained three-branch network structure model, wherein the three-branch network structure model includes: a detail branch, a context branch, and a boundary branch; The detail branch is used to parse the detail information of the processed feature map to obtain the detail feature map of the robot vision picture; the boundary branch is used to extract the pixel change information of the processed feature map, identify the boundary, and obtain the boundary feature map of the robot vision picture; the context branch is used to extract the deep semantic information of the processed feature map and aggregate the long-range information to obtain the context feature map of the robot vision picture; the obtained detail feature map, boundary feature map and context feature map are balanced to obtain a balanced feature map; The balanced feature map is further processed with semantic information to obtain the semantic segmentation result of the robot vision image; The method of using the context branch to extract the deep semantic information of the processed feature graph and aggregate the long-range information to obtain the context feature graph of the robot visual image includes: aggregating the feature graph F9 after the processed feature graph F1 is processed by three times of downsampling and semantic enrichment to obtain the context feature graph F11; The method of using the detail branch to parse the detail information of the processed feature map to obtain the detail feature map of the robot visual image includes: subjecting the processed feature map F1 to fine-grained residual enriched semantic processing to obtain a feature map F2, and fusing the feature map F2 with the feature map obtained after the processed feature map F1 is processed once and twice by the context branch through downsampling enriched semantic processing to obtain a detail feature map F8; The method of using the boundary branch to extract pixel change information of the processed feature map, identify the boundary, and obtain the boundary feature map of the robot vision image includes: subjecting the processed feature map F1 to coarse-grained residual semantic enrichment processing to obtain a feature map F4, and sequentially adding the feature map F4 to the feature map obtained after the processed feature map F1 is processed once and twice by downsampling semantic enrichment processing through the context branch to obtain a boundary feature map F10; The feature map F2 is sequentially fused with the feature map F1 obtained after one and two downsampling and semantic enrichment processing through the context branch to obtain the detail feature map F8, which specifically includes the following steps: The feature map F2 is fused with the feature map F3 obtained after the processed feature map F1 is processed by downsampling once through the context branch and semantic enrichment, and the feature map F5 is obtained. The corresponding expression is as follows: σ=Sigmoid(f d (F2)f c (F3)), F5=(1-σ)F2+σF3, Where: σ is the output of the Sigmoid function, F5 is the feature map output after the fusion of feature map F2 and feature map F3; f d is a combination of convolution, batch normalization, and ReLu operations, f d The convolution kernel size is 3 and the stride is 1; f c is a combination of convolution, batch normalization, ReLu, and upsampling operations, f c The convolution kernel size is 3 and the stride is 1; The feature map F5 is fused with the feature map F6 obtained after the processed feature map F1 is processed by downsampling twice through the context branch and enriching the semantics, to obtain the detail feature map F8. The corresponding expression is as follows; σ=Sigmoid(f d (F5)f c (F6)), F8=(1-σ)F5+σF6.

2. The semantic segmentation method of the three-branch network structure according to claim 1, characterized in that: The processing of the feature map F9 after the feature map F1 is processed by three downsampling enrichment semantics processing in sequence to obtain the context feature map F11 includes: The feature map F9 is input into four pooling layers respectively, and four feature maps are output through 1*1 convolution and upsampling operations; The sum of the four feature maps and the feature map F21 obtained by the 1*1 convolution of the feature map F9 is respectively subjected to a 3*3 convolution operation; Concatenate the 4 values ​​obtained from the 3*3 convolution operation with the feature map F21; Add the value after the connection operation to the feature map F21 to obtain the context feature map F11 output by the aggregation process; Among them: the four pooling layers are average pooling with a stride of 2 and a window size of 5; average pooling with a stride of 4 and a window size of 9; average pooling with a stride of 8 and a window size of 19 and a global pooling layer.

3. The semantic segmentation method of the three-branch network structure according to claim 2, characterized in that: The obtained detail feature map, boundary feature map and context feature map are balanced to obtain the expression of the balanced feature map as follows: σ=Sigmoid(F10), Out=f out ((1-σ)F11+σF8), Where: σ is the output of the Sigmoid function, F8 is the detail feature map, F10 is the boundary feature map, F11 is the context feature map, and Out is the output balanced feature map; f out It is a combination of convolution, batch normalization and ReLu operations, with a kernel size of 3 and a stride of 1.

4. The semantic segmentation method of the three-branch network structure according to claim 1, characterized in that: The training method of the three-branch network structure model comprises the following steps: The loss function L1 is set according to the boundary feature map, the loss function L2 is set according to the context feature map, and the loss function L3 is set by combining the context feature map and the boundary feature map; The total loss function is constructed based on the loss function L1, loss function L2, and loss function L3: Loss=λ1L1+λ2L2+λ3L3, Among them: L1 is the weighted binary cross entropy loss function set for the boundary branch, L2 is the cross entropy loss function set for the context branch, and L3 is the boundary-aware loss function; λ1, λ2, λ3 are L1, L2, L3 weight parameters respectively; Where: t is the predefined threshold, b i is the output of the boundary header; S i,c is the segmentation truth value of the i-th pixel of category c; is the prediction result of the i-th pixel of category c, The constructed three-branch network structure model is iteratively trained using a robot vision image dataset until the total loss function converges or reaches an upper limit on the number of iterations.

5. The semantic segmentation method of the three-branch network structure according to claim 1, characterized in that: The fine-grained residual enriched semantic processing includes: convolution, batch normalization, ReLu, convolution, batch normalization, residual connection, ReLu, convolution, batch normalization, ReLu, convolution, batch normalization, residual connection processing steps in sequence; wherein the convolution kernel is 3 and the stride is 1; The coarse-grained residual enriched semantic processing includes: convolution, batch normalization, ReLu, convolution, batch normalization, residual connection, ReLu processing steps in sequence, wherein the convolution kernel is 3 and the stride is 1; The semantic information fine processing includes: batch normalization, ReLu, 3*3 convolution, batch normalization, ReLu and 1*1 convolution processing steps in sequence.

6. The semantic segmentation method of the three-branch network structure according to claim 1, characterized in that: Downsampling and semantic enrichment processing is performed on the original feature map to obtain a processed feature map, including: The original feature map is sequentially subjected to the steps of convolution, batch normalization, ReLu, convolution, batch normalization, downsampling, residual connection, ReLu, convolution, batch normalization, ReLu, convolution, batch normalization, and residual connection to obtain a processed feature map having a resolution of 1 / 8 of the original feature map; Among them, the convolution kernel is 3 and the stride is 1.

7. A semantic segmentation system of a three-branch network structure based on the semantic segmentation method of the three-branch network structure according to claim 1, characterized in that: include: An extraction module is used to extract features from robot vision images using a cascaded residual backbone network to obtain the corresponding original feature map; A downsampling module is used to perform downsampling and semantic enrichment processing on the original feature map to obtain a processed feature map; A processing module is used to input a processed feature map into a trained three-branch network structure model, wherein the three-branch network structure model includes: a detail branch, a context branch and a boundary branch; the detail branch is used to parse the detail information of the processed feature map to obtain a detail feature map of a robot vision picture; the boundary branch is used to extract pixel change information of the processed feature map, identify boundaries, and obtain a boundary feature map of the robot vision picture; the context branch is used to extract deep semantic information and aggregate long-range information of the processed feature map to obtain a context feature map of the robot vision picture, and the obtained detail feature map, boundary feature map and context feature map are subjected to balancing processing to obtain a balanced feature map; the balanced feature map is further subjected to fine processing of semantic information to obtain a semantic segmentation result of the robot vision picture.

8. A semantic segmentation device with a three-branch network structure, characterized in that: include: Processor and storage media; The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the semantic segmentation method of the three-branch network structure according to any one of claims 1 to 6.

Images