An image semantic segmentation algorithm and system based on multi-channel deep weighted aggregation

Abstract

The present invention provides an image semantic segmentation algorithm and system based on multi-channel deep weighted aggregation, wherein the image semantic segmentation algorithm based on multi-channel deep weighted aggregation includes the following steps: S1, extracting semantic features with clear category information, transitional semantic features between low-level semantics and high-level semantics, and semantic features of contextual logical relationships in the image through low-level semantic channels, auxiliary semantic channels, and high-level semantic channels; S2, fusing the three different semantic features obtained in step S1 through weighted aggregation to obtain the global semantic information of the image; S3, calculating the loss function together with the semantic features output by each semantic channel in step S1 and the global semantic information in step S2 for training. The image semantic segmentation algorithm based on multi-channel deep weighted aggregation in the present invention can solve the problem that the traditional image semantic segmentation algorithm cannot achieve a trade-off between segmentation accuracy and speed, and has stronger robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to an image semantic segmentation algorithm and system based on multi-channel depth weighted aggregation. Background Art

[0002] In recent years, with the continuous development of deep learning technology, image semantic segmentation technology has also developed to a new stage, and various semantic segmentation algorithms have emerged. Among them, the image semantic segmentation algorithm is an important intelligent perception technology. Its task is to assign semantic labels to each pixel, separate the pixel areas of different objects in the image, and label the category of each area. Traditional semantic segmentation algorithms have many defects, such as slow reasoning speed, low semantic segmentation accuracy, and inability to run in real time on intelligent mobile robots.

[0003] For most semantic segmentation algorithms, how to maintain a balance between the speed and accuracy of the algorithm is still a key issue. In traditional semantic segmentation algorithms, some methods limit the image size or reduce the computational complexity of the network by pruning redundant channels of the network to improve the algorithm reasoning speed, but the segmentation accuracy of these semantic segmentation algorithms is low; there are also some methods that use U-shaped structures to operate on high-resolution feature maps to improve segmentation accuracy, but this will greatly increase the computational complexity and make the algorithm reasoning speed too slow. Obviously, the above traditional semantic segmentation algorithms cannot strike a good balance between accuracy and speed.

[0004] Considering that most current semantic segmentation algorithms have limited learning capabilities, resulting in the inability to significantly improve the overall performance of the algorithms, some researchers have proposed ways to improve the learning capabilities of semantic segmentation algorithms. Among them, neuroevolution methods are the most significant, such as neuroevolutionary convolutional neural networks ConvGP (Convolutional Genetic Programming), generative adversarial networks EGAN (Evolutionary Generative Adversarial Networks), autoencoders EvoAE (Evolutionary AutoEncoder), long short-term memory networks CoDeepNEAT (Convolution Deep Neuro Evolution of Augmenting Topologies), deep reinforcement learning WANN (Weight Agnostic Neural Network) and other typical models. These models can greatly improve the learning ability of the algorithm, but are still greatly limited in inference speed and cannot fundamentally solve the problem of balancing algorithm speed and accuracy.

[0005] Some researchers have also proposed some improvement measures: for example, the BiSeNet semantic segmentation algorithm uses a dual-branch structure combined with spatial information to maximize the segmentation accuracy of the algorithm while ensuring a certain speed, and has achieved certain results. However, the mutual independence between branches in the dual-branch structure limits the learning ability of the algorithm, making it unable to better learn the global semantic features of the image, making the improvement of segmentation accuracy very limited.

[0006] The DFANet semantic segmentation algorithm uses a multi-branch structure and enables information exchange between branches to extract image semantic features. While ensuring the algorithm's reasoning speed, it improves the algorithm's segmentation accuracy to a certain extent. However, the multi-branch structure is the same, so it can only extract one kind of semantic information, and it is unable to better obtain global semantic and context information features, which limits the algorithm's segmentation accuracy.

[0007] In summary, although there are many semantic segmentation algorithms based on deep learning, these algorithms can hardly guarantee the balance between image segmentation accuracy and reasoning speed. To this end, the present invention proposes a new semantic segmentation network framework to achieve a trade-off between segmentation accuracy and reasoning speed of image semantic segmentation algorithms. Summary of the invention

[0008] In response to the technical problems existing in the prior art, the purpose of the present invention is to provide an image semantic segmentation algorithm and system based on multi-channel deep weighted aggregation, which can solve the problem that traditional image semantic segmentation algorithms cannot achieve a trade-off between segmentation accuracy and speed, and has stronger robustness.

[0009] The image semantic segmentation algorithm based on multi-channel depth weighted aggregation provided by the present invention comprises the following steps:

[0010] S1, through the low-level semantic channel, auxiliary semantic channel and high-level semantic channel, respectively extract the semantic features with clear category information in the image, the transitional semantic features between low-level semantics and high-level semantics, and the semantic features of the contextual logical relationship in the image;

[0011] S2, fusing the three different semantic features obtained in step S1 through weighted aggregation to obtain the global semantic information of the image;

[0012] S3, the semantic features output by each semantic channel in step S1 and the global semantic information in step S2 are used together to calculate the loss function for training.

[0013] Preferably, in step S1: a shallow convolutional structure network is used to construct a low-level semantic channel for extracting low-level semantic information;

[0014] A deep separable convolutional structure network is used to construct an auxiliary semantic channel, and the transitional semantic information obtained by the auxiliary semantic channel is fed back to the high-level semantic channel;

[0015] A deep convolutional structure network is used to construct a high-level semantic channel to extract high-level semantic information.

[0016] Preferably, the process of extracting low-level semantic information by a shallow convolutional structure network includes:

[0017] LS(I H*W )=S3(S2(S1(I H*W )))

[0018] Among them, LS(x) is the low-level semantic information extraction process, I H*W is the input image matrix and S is the convolution step size.

[0019] Preferably, the separable convolutional structure network includes three auxiliary modules connected in series and an attention mechanism module at the tail, and each module transmits the extracted auxiliary semantic information of different levels to the high-level semantic channel;

[0020] The process of extracting transition semantic information through the auxiliary semantic channel includes:

[0021] AS(K m*n )=Atten(Aux(Aux(Aux(K m*n ))))

[0022] Among them, AS(x) is the transition semantic information extraction process, Km*n is the feature matrix of the input auxiliary semantic channel, Aux(x) is the auxiliary module, and Atten(x) is the attention mechanism module;

[0023] The auxiliary semantic information output by each stage of the auxiliary semantic channel includes:

[0024] Aux1(K m*n )=Aux(K m*n )

[0025] Aux2(K m*n )=Aux(Aux1(K m*n ))

[0026] Aux3(K m*n )=Aux(Aux2(K m*n ))

[0027] Aux1'(K m*n )=Up(AS(K m*n ),4)

[0028] Among them, Up(x,k) is the upsampling of the attention mechanism module to the high-level semantic channel, x is the input, k is the upsampling multiple, and the selected value is 4.

[0029] Preferably, the high-level semantic channel includes a startup module, a feature aggregation module and a semantic nesting module, and the process of extracting high-level semantic information includes:

[0030] The deep semantic information of the image is extracted by fusing the auxiliary semantic information provided by the starting module and the feature aggregation module with the auxiliary semantic channel. Then, the semantic nesting module is used to integrate the deep semantic information of the auxiliary semantic channel with the high-level semantic channel to obtain the high-level semantic information.

[0031] Preferably, the startup module includes two different downsampling methods, namely, convolution and maximum pooling, and the two output results are connected and then output by convolution; the feature aggregation module includes two feature aggregation submodules, and the number of feature layers is deepened by deep convolution. The aggregation process of the feature aggregation module includes:

[0032] FGB(K′ m*n ,s1,s′1)=FG(FG(K′ m*n ,s1),s′1)

[0033] Among them, FGB(x) is the reasoning process of the feature aggregation module, FG(x,s1) represents the reasoning process of the feature aggregation submodule, and K′ m*n is the input feature matrix, the step size s1=2, s1=1;

[0034] The reasoning process of the feature aggregation module of the three auxiliary modules in the auxiliary semantic channel includes:

[0035] FGB1(K m*n )=FGB(ST(K m*n )+Aux1(K m*n )+Aux1'(K m*n ),s1,s′1)

[0036] FGB2(K m*n )=FGB(FGB1(K m*n )+Aux2(K m*n ),s1,s′1)

[0037] FGB3(K m*n )=FGB(FGB2(K m*n )+Aux3(K m*n ),s1,s′1)

[0038] The semantic embedding module adopts global average pooling and skip connection structure to deeply fuse the auxiliary semantic information provided by the high-level semantic channel and the auxiliary semantic channel;

[0039] The process of extracting semantic information from high-level semantic channels includes:

[0040] AS(K m*n )=SEB(FGB3(K m*n ))

[0041] Among them, AS(x) is the semantic information extraction process of the high-level semantic channel, and SEB(x) is the reasoning process of the semantic nested module.

[0042] Preferably, the fusion of semantic features in step S2 includes weighted aggregation and fusion of three types of semantic features.

[0043] Weighted aggregation includes: updating the low-level semantic channel, auxiliary semantic channel and high-level semantic channel to obtain the weighted weights ε1, ε2 and ε3 of the semantic features respectively. The weight update is adaptively determined by the MIoU = {MIoU1, MIoU2, MIoU3} measured by the three types of semantic features in the validation set. The network weights of each channel are used to obtain MIoU on the validation set, and then the MIoU value is used to update the weighted weights of the three types of semantic features according to the following formula, including:

[0044]

[0045]

[0046]

[0047] The weighting process of three types of semantic features includes:

[0048]

[0049] Wherein, Up is an upsampling with a magnification of 2, Up(x) represents the upsampling process, Conv is a convolution operation, and the calculation of sigmoid(x) includes:

[0050]

[0051] Preferably, the fusion of the three types of semantic features includes multiplying the weighted results obtained from the semantic channels pairwise and then performing a 3*3 convolution with a step size of 1. The pairwise aggregation process of the three types of semantic features includes:

[0052]

[0053] Among them, SA1(x), SA2(x) and SA3(x) are the pairwise aggregation processes of three types of semantic features respectively;

[0054] The compound process of the three-way semantic features obtained by the three types of semantic channels includes:

[0055] MSA(x)=SA1(x)+SA2(x)+SA3(x)

[0056] Among them, MSA(x) is a composite process of three-way semantic features.

[0057] Preferably, in step S3, the semantic features output by each channel are upsampled and the loss is calculated together with the global semantic information of the image to perform stochastic gradient descent learning, and the loss function includes the calculation of weights, the calculation of loss functions, and the determination of loss functions in a comprehensive training mechanism, wherein the calculation of weights includes:

[0058]

[0059] Among them, α is the weight, n is the number of categories;

[0060] The calculation of the loss function includes:

[0061] CED(x,y)=CEL(x,y)+α*DL(x,y)

[0062] Among them, x is the predicted data, y is the real data, CEL(x,y) is the Cross-Entropy loss function, and DL(x,y) is the Dice loss function;

[0063] The determination of the loss function in the comprehensive training mechanism includes:

[0064]

[0065] Among them, T is the true label, P re Output labels for global semantic information. A1, A2, and A3 are the output labels of low-level semantic channel, auxiliary semantic channel, and high-level semantic channel, respectively.

[0066] The present invention also provides an image semantic segmentation system, which adopts the above-mentioned image semantic segmentation algorithm based on multi-channel deep weighted aggregation, including: a three-channel semantic representation model, three types of semantic weighted aggregation modules and an enhanced training module;

[0067] The three-channel semantic representation model includes a low-level semantic channel, an auxiliary semantic channel and a high-level semantic channel; the auxiliary semantic channel includes an auxiliary module and an attention mechanism module, and the high-level semantic channel includes a startup module, a feature aggregation module and a semantic nesting module.

[0068] In view of the problem that traditional image semantic segmentation algorithms cannot achieve a trade-off between segmentation accuracy and speed, the present invention provides a new high-precision real-time image semantic segmentation algorithm based on Multi-Channel Deep Weighted AggregationNet (MCDWA Net, abbreviated as MCDWA_Net).

[0069] First, the multi-channel idea is introduced through a three-channel semantic representation model, which mainly includes three different semantic channels: low-level semantic channel, auxiliary semantic channel and high-level semantic channel, which are used to extract three types of complementary semantic information. The low-level semantic channel outputs semantic features with clear category information in the image; the auxiliary semantic channel extracts transition information between low-level semantics and high-level semantics, and realizes multi-layer feedback to the high-level semantic channel to ensure the rapidity and accuracy of high-level semantic channel extraction; the high-level semantic channel obtains the semantic features of the contextual logical relationship in the image.

[0070] Secondly, through the three-category semantic feature weighted aggregation module, the complementary semantic features output by the three channels can be weighted and deeply fused to output global semantic features, thereby greatly improving the segmentation accuracy of the network.

[0071] Finally, the enhanced training module improves the defects of the two loss functions and combines them to strengthen the feature representation in the training phase and enhance and improve the training speed.

[0072] Compared with traditional semantic segmentation algorithms, the image semantic segmentation algorithm in the present invention has higher scene segmentation accuracy, faster reasoning speed, stronger adaptability to various complex environments, and better practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0074] Figure 1 It is a logical framework diagram of the image semantic segmentation algorithm based on multi-channel depth weighted aggregation of the present invention;

[0075] Figure 2 It is a schematic diagram of the low-level semantic channel network structure;

[0076] Figure 3 This is a schematic diagram of the auxiliary semantic channel network structure;

[0077] Figure 4 It is a schematic diagram of the high-level semantic channel network structure;

[0078] Figure 5 This is a schematic diagram of the startup module network structure;

[0079] Figure 6 This is a schematic diagram of the network structure of the feature aggregation submodule;

[0080] Figure 7 This is a schematic diagram of the semantic nested module network structure;

[0081] Figure 8 Schematic diagram of weighted aggregation module for three types of semantic features;

[0082] Fig. 9 It is a schematic diagram of the enhanced training module;

[0083] Fig.10 This is a curve diagram of the change in the model training loss value in Example 1;

[0084] Fig.11 This is a curve diagram of the MIoU value change in the model training in Example 1;

[0085] Fig.12 This is the effect diagram of ablation experiment;

[0086] Fig.13 The actual segmentation effect diagram of the six semantic segmentation algorithms in Example 1;

[0087] Fig.14 This is a curve diagram of the change in the model training loss value in Example 2;

[0088] Fig.15 This is a curve diagram of the MIoU value change in the model training in Example 2;

[0089] Fig.16 This is a diagram showing the segmentation effects of the six semantic segmentation algorithms in Example 2. DETAILED DESCRIPTION

[0090] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0091] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0092] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0093] See also Figure 1 The image semantic segmentation algorithm based on multi-channel deep weighted aggregation provided by the present invention has the following main steps: 1) using a shallow convolutional network structure to construct a low-level semantic channel for extracting semantic features with clear category information in the image; 2) using deep separable convolution to construct a deeper auxiliary semantic channel for extracting transition information between low-level semantics and high-level semantics and realizing multi-layer feedback on the high-level semantic channel; 3) using a deep convolutional structure to construct a high-level semantic channel to obtain the semantic features of the contextual logical relationship in the image.

[0094] Then, a three-category semantic feature weighted aggregation module is designed. This module can deeply fuse the complementary semantic features output by the three channels after weighting, and output the global semantic features, thereby greatly improving the segmentation accuracy of the network.

[0095] Finally, an enhanced training mechanism is introduced to strengthen the feature representation in the training phase and enhance and improve the training speed.

[0096] Compared with traditional semantic segmentation algorithms, it has stronger robustness and can achieve a trade-off between speed and accuracy of image semantic segmentation.

[0097] The image semantic segmentation algorithm based on multi-channel deep weighted aggregation proposed in the present invention, compared with the traditional semantic segmentation algorithm, introduces the multi-channel idea and designs a three-channel semantic representation model for extracting three different types of semantic information in the image. The model can extract various types of semantic information in the image to the greatest extent, and distribute the semantic information extraction task to three channels to complete, which greatly reduces the algorithm reasoning time and improves the algorithm running speed; in order to better integrate the three types of local semantic information, the present invention provides three types of semantic feature weighted aggregation modules, so that the three types of semantic information are weighted and then fused in pairs, and finally all of them are fused together to fuse into global semantic information; in addition, in order to improve the algorithm training speed, the algorithm model can be trained faster by enhancing the training module.

[0098] The present invention adopts the following technical solutions and implementation steps:

[0099] The three-channel semantic features are low-level semantic channel, auxiliary semantic channel and high-level semantic channel.

[0100] The low-level semantic channel is responsible for extracting semantic features with clear category information in the image. Such semantic information can only be fully displayed in the high-resolution feature map. Therefore, the present invention extracts low-level category semantic information through a shallow structure convolutional network, such as Figure 2 As shown in Figure 2, this simple shallow structure convolutional network can not only extract high-resolution feature maps, but also greatly reduce the inference time.

[0101] Assuming the input image matrix is ​​IH*W and the number of channels is 3, the extraction of low-level semantic information can be expressed as the process of S1-S3 in Table 1.

[0102] In Table 1, H and W are the height and width of the input image, respectively, Conv represents the convolution operation, k is the convolution kernel size, Cout is the number of output image channels, and s is the convolution step size.

[0103] Table 1. Low-level semantic channel semantic information extraction process

[0104]

[0105] Assume that function LS(x) is the low-level semantic channel information extraction process, then the low-level semantic information extraction process is shown in formula (1):

[0106] LS(I H*W )=S3(S2(S1(I H*W ))) (1)

[0107] As can be seen from Table 1, the low-level semantic channel will eventually output the low-level semantic feature LS (I H*W), which can better express the semantic features with clear category information in the image.

[0108] The auxiliary semantic channel is responsible for extracting the transitional semantic information between low-level semantics and high-level semantics in the image, and providing it to the high-level semantic channel to assist the high-level semantic channel in extracting image context information, thereby playing an auxiliary role.

[0109] The present invention adopts deep separable convolution (DSC) and designs an auxiliary module (Aux) that can not only improve the running speed but also extract deeper feature information. The three auxiliary modules (Aux) are connected in series, and a fully connected attention mechanism module (Atten) is added at the end to retain the largest receptive field and output more complete auxiliary feature information. Finally, the auxiliary semantic features extracted at each stage are passed to the high-level semantic module to construct an auxiliary channel. The schematic diagram of its network structure is shown in the figure. Figure 3 shown.

[0110] like Figure 3 In the figure, the Aux module combines the results of three layers of depthwise separable convolution (DSC) and one layer of 3*3 convolution with a stride of 2 and outputs them. The Atten module first converts the input into a matrix structure of n*1000*1*1 (n is the number of categories), then performs a 1*1 convolution with a stride of 1, and finally restores it to the original input shape and merges it with the input before outputting it.

[0111] Assume that the feature matrix of the input auxiliary semantic channel is Km*n, the Aux module is represented by the Aux(x) function, the Atten module is represented by Atten(x), and the auxiliary semantic channel information extraction process is represented by AS(x). The auxiliary semantic channel information extraction process can be expressed as follows:

[0112] AS(K m*n )=Atten(Aux(Aux(Aux(K m*n )))) (2)

[0113] Further, by Figure 3 It can be seen that the auxiliary semantic channel finally outputs the transition semantic feature AS(K m*n ).

[0114] Based on the above analysis, let the upsampling operation be Up(x, k) (where x is the input, k is the upsampling multiple, k=4 here), and the auxiliary semantic information output by each stage of the auxiliary semantic channel can be expressed as:

[0115] Aux1(K m*n )=Aux(K m*n ) (3)

[0116] Aux2(Km*n )=Aux(Aux1(K m*n )) (4)

[0117] Aux3(K m*n )=Aux(Aux2(K m*n )) (5)

[0118] Aux1'(K m*n )=Up(AS(K m*n ),4) (6)

[0119] In summary, it can be seen that the auxiliary semantic channel feeds back the auxiliary semantic information of different levels output by each stage to the high-level semantic channel, so that the high-level semantic channel can more accurately extract the contextual logical relationship in the image and speed up the extraction speed and accuracy.

[0120] The high-level semantic channel is responsible for obtaining the semantic features of the contextual logical relationship in the image. The high-level semantic channel extracts the deep semantic information of the image by fusing the auxiliary semantic information provided by the auxiliary semantic channel through the start module (Start) and the feature aggregation module (CGB). Finally, the semantic nesting module (SEB) is used to integrate the deep semantic information of the auxiliary semantic channel with the high-level semantic channel to complete the information extraction process of the high-level semantic channel. The network structure diagram is shown in the figure. Figure 4 shown.

[0121] Start module: In the high-level semantic channel, the start module is used as the first stage, such as Figure 5 As shown in the figure, it uses two different downsampling methods, convolution and maximum pooling, and then connects their output results and convolves them to enhance the feature expression ability. Let the function ST(x) be the reasoning process of the startup module.

[0122] Feature aggregation module (FGB): After the startup module is the feature aggregation module, the feature aggregation module consists of two feature aggregation sub-modules. The network structure diagram of the feature aggregation sub-module is as follows: Figure 6 As shown in Figure 2, deep convolution is used to deepen the number of feature layers, extract deeper semantic information, aggregate semantic features more effectively, and output deeper features.

[0123] Assuming that FG(x,s1) is used to represent the feature aggregation submodule reasoning process, and FGB(x) is used to represent the feature aggregation module reasoning process, the aggregation process is shown in formula (7):

[0124] FGB(K' m*n ,s1,s′1)=FG(FG(K′ m*n ,s1),s′1) (7)

[0125] Among them, K' m*nis the input feature matrix, with step size s1=2, s′1=1.

[0126] Further combining formulas (3), (4), (5), (6) and (7), the reasoning process of FGB1, FGB2 and FGB3 can be expressed as follows:

[0127] FGB1(K m*n )=FGB(ST(K m*n )+Aux1(K m*n )+Aux1'(K m*n ),s1,s′1) (8)

[0128] FGB2(K m*n )=FGB(FGB1(K m*n )+Aux2(K m*n ),s1,s′1) (9)

[0129] FGB3(K m*n )=FGB(FGB2(K m*n )+Aux3(K m*n ),s1,s′1) (10)

[0130] Semantic Embedding Module (SEB): At the final stage of the high-level semantic channel is a semantic embedding module, which uses global average pooling and skip connection structure to deeply fuse the auxiliary semantic information provided by the high-level semantic channel and the auxiliary semantic channel, so as to more effectively embed the global context information, such as Figure 7 As shown. Let SEB(x) represent the reasoning process of this module, and AS(x) be the high-level semantic channel semantic information extraction process. Then, combined with formula (10), the reasoning process of AS(x) can be expressed as follows:

[0131] AS(K m*n )=SEB(FGB3(K m*n )) (11)

[0132] In summary, the high-level semantic features AS(K m*n ).

[0133] The above three types of semantic information are all local semantic features of the image and are complementary. Therefore, the three need to be fused to obtain the global semantic information of the image. The present invention realizes the fusion of the three types of semantics by adopting a feature weighted aggregation method.

[0134] The specific fusion principle is as follows Figure 8 As shown in the figure, when an H*W*3 image is input, semantic features are first extracted through three types of semantic channels, and then the weighted aggregation process is as follows:

[0135] Analysis of weighted aggregation principle

[0136] Assume that the weights of the three types of semantic features are ε1, ε2 and ε3 respectively, and their initial values ​​are all set to 1. The change of weights during operation is adaptively determined by the MIoU = {MIoU1, MIoU2, MIoU3} measured by the three types of semantic features in the validation set. The specific update rule is to use the current network weights of each channel to obtain MIoU on the validation set, and then use the MIoU value to update ε1, ε2 and ε3 according to the following formulas (14) to (16).

[0137]

[0138]

[0139]

[0140] like Figure 8 Down is a 3*3 convolution with a stride of 2, and Up is an upsampling with a magnification of 2. Let Up(x) represent the upsampling process. The weighting process of the three types of semantic features is as follows:

[0141]

[0142] In the formula,

[0143]

[0144] Three types of semantic feature aggregation

[0145] like Figure 8 middle It means that the two results are multiplied and then a 3*3 convolution with a step size of 1 is performed. Let the convolution process be represented by Conv1(x), the aggregation process of the three types of semantic features is represented by SA1(x), SA2(x) and SA3(x) respectively, and the final composite process is represented by MSA(x). From formula (15), we can get:

[0146]

[0147] The final composite process of the three-way semantic features can be expressed as follows:

[0148] MSA(x)=SA1(x)+SA2(x)+SA3(x) (18)

[0149] In summary, formulas (15) to (18) complete the reasoning process of weighted aggregation of three types of semantic features.

[0150] In order to further improve the training speed and training effect, the present invention designs a new enhanced training module, which can enhance the feature representation in the training stage. Fig. 9As shown in the figure, when training the MCDWA_Net network, the output features of each channel are upsampled and the loss is calculated together with the final output result of the network for stochastic gradient descent learning.

[0151] In traditional semantic segmentation, the Cross-Entropy loss function is generally used for gradient descent learning, as shown in formula (19):

[0152]

[0153] Among them, p is the real data, q is the predicted data, and C is the number of categories.

[0154] However, when using the Cross-Entropy loss function for semantic segmentation, the weight update will be affected by errors. For example, the foreground pixels in the image are much smaller than the background pixels, and p in formula (19) i =0 part will dominate, making the trained model biased towards the background.

[0155] To this end, another loss function Dice loss is introduced as shown in formula (20):

[0156]

[0157] In formula (20), X is the predicted data and Y is the real data. In general, the use of Dice loss function is not conducive to back propagation and makes the training process unstable, so this loss function is not widely used.

[0158] However, through experiments, it is found that the Dice loss function has a characteristic that when the number of foreground pixels and background pixels in the image is unbalanced, the function can just make up for the defects of the Cross-Entropy loss function and eliminate the bias effect. To this end, the present invention combines the two and designs a joint loss function based on Cross-Entropy and Dice loss, referred to as CEDice loss function.

[0159] Let x be the predicted data, y be the real data, n be the number of categories, and α be the weight, then the formula is as follows:

[0160]

[0161] Then the CEDice loss function can be expressed as follows:

[0162] CED(x,y)=CEL(x,y)+α*DL(x,y) (22)

[0163] CEL(x, y) is the Cross-Entropy loss function, and DL(x, y) is the Dice loss hinge loss function. It can be seen from equations (21) and (22) that when the number of segmentation categories in semantic segmentation is 1, it is equivalent to a binary classification. At this time, there is no need to use the Dice loss function. Therefore, only the Cross-Entropy loss function works. When the number of semantic segmentation categories is larger, there will be a situation where the foreground pixels in the image are much smaller than the background pixels, that is, the number of foreground pixels and background pixels in the image is unbalanced. At this time, the larger the weight of the Dice loss function is, the greater the impact of pixel imbalance will be.

[0164] In summary, the loss function in the comprehensive training mechanism of the present invention is replaced by the CEDice loss function, and the final loss function can be set as:

[0165]

[0166] In formula (23), T is the true label, Pre is the MCDWA_Net network output label, A1, A2 and A3 are the output labels of the low-level semantic channel, auxiliary semantic channel and high-level semantic channel respectively.

[0167] The following describes the technical solution in the present invention for different application environments. The hardware environment of the simulation experiment platform in the present invention is Intel Core i7-10750 CPU, @2.6GHz, x6 cores, 16GB RAM, and the operating environment of the experiment is Pytorch1.6.0.

[0168] Example 1

[0169] In this embodiment, experiments are mainly conducted on the Cityscapes street view dataset.

[0170] In the simulation experiment, the Cityscapes street scene dataset was first used for verification. This dataset has 18 categories of street scenes and 1 background (the total number of categories is 19), and 3475 pictures and their labels in the dataset were used for training, verification and testing. The number of pictures in the training set, verification set and test set is 2975, 300 and 200 respectively. Afterwards, an actual scene dataset of the laboratory corridor was produced. This dataset has 7 corridor scene classes and 1 background class (the total number of categories is 8), a total of 350 pictures and their labels, 300 of which are used for training the model, 30 for verifying the model, and 20 for testing the model. In order to fully demonstrate the effectiveness of the algorithm of the present invention, it was compared and analyzed with BiSeNet, BiSeNetV2, DFANet, Deeplab V3 and ShuffleNet V2 algorithms.

[0171] 1) Model training

[0172] When training the MCDWA_Net model with the Cityscapes street view dataset, the training batch Batch_size is set to 4, the number of categories Num_classes is set to 19, the number of iterations Epoches is set to 500, the learning strategy adopts the stochastic gradient descent (SGD) algorithm, and its momentum is set to 0.9, the initial learning rate is set to 0.05, and the weight decay rate is set to 0.0001.

[0173] When training the model, the enhanced training module proposed in the present invention is used to apply the three types of semantic features extracted from each channel to calculate the loss function using formula (23), and the stochastic gradient descent method is used for training and learning. The change of the loss value during the training process is as follows: Fig.10 shown.

[0174] 2) Model performance evaluation indicators

[0175] First, the Mean Intersection over Union (MIoU) is introduced to evaluate the accuracy of the model. MIoU evaluates the accuracy of the model in segmenting objects. The higher the MIoU value, the better the object segmentation effect. The calculation method is shown in formula (23):

[0176]

[0177] Where k is the number of categories, FTi represents the number of samples in the ith category that are predicted incorrectly and predicted to be true, FFi represents the number of samples in the ith category that are predicted incorrectly and predicted to be false, and TTi represents the number of samples in the ith category that are predicted correctly.

[0178] The MIoU value changes during the training process of the algorithm of the present invention on the Cityscapes street view dataset are as follows: Fig.11 shown.

[0179] 3) Model ablation experiment

[0180] Ablation experiments were completed on the Cityscapes street view dataset to verify the effectiveness of each module in the network model, and the Cityscapes street view validation set was used to evaluate the algorithm.

[0181] Table 2 and Fig.12This is the experimental result of MCDWA_Net on the Cityscapes street view dataset. The first three rows in Table 2 show the segmentation accuracy and computational complexity when only one channel is used. The low-level semantic channel cannot obtain the contextual logical relationship features of the image, the auxiliary semantic channel can only extract transitional semantic information, and the high-level semantic channel contains the contextual logical relationship semantic features of the image, but there is no clear category information. Therefore, these three semantic channels cannot obtain the complete semantic information of the image. The three types of semantic information must be fused to express the complete semantic information of the image, and the fusion method results also play a crucial role. For example, the direct addition fusion effect in the fourth row of Table 2 is obviously much worse than the semantic feature weighted fusion effect in the fifth row. However, it can also be seen that no matter what fusion method is used, the segmentation effect after fusion is significantly better than the segmentation effect of each channel, indicating that the semantic information extracted by the three channels is local and complementary.

[0182] In addition, the model training method also affects the segmentation effect of the model. The segmentation accuracy of the model trained by the ordinary training method is 0.6% lower than that of the enhanced training method of the present invention. Therefore, to a certain extent, the enhanced training method designed by the present invention can also improve the segmentation accuracy of the training model. In summary, after weighted aggregation of the three types of complementary semantic information, the semantic segmentation effect of the image is better, the segmentation accuracy is higher, and the context information expression is clearer.

[0183] Table 2 Ablation experiments of MCDWA_Net on Cityscapes street view dataset

[0184]

[0185]

[0186] 4) Model performance comparison experiment

[0187] The BiSeNet, BiSeNetV2, DFANet, Deeplab V3 and ShuffleNet V2 algorithms were trained on the Cityscapes street view dataset to obtain corresponding models, and the traditional training strategy was used for training.

[0188] Table 3 Performance comparison of MCDWA_Net algorithm and advanced algorithms on Cityscapes street view dataset

[0189]

[0190] Table 3 shows the performance comparison results of MCDWA_Net and five relatively latest semantic segmentation algorithms. It can be clearly seen from Table 3 that the segmentation accuracy of the MCDWA_Net algorithm proposed in the present invention is higher than that of the other five algorithms, but its computational complexity is more complex than that of BiSeNet, BiSeNetV2, and DFANet algorithms, so its reasoning speed is slightly slower than these three algorithms, and it is better than Deeplab V3 and ShuffleNet V2 algorithms in both algorithm accuracy and reasoning speed.

[0191] The specific segmentation effects of the above 6 algorithms are as follows Fig.13 As shown in Table 3 and Fig.13 It can be seen that although the algorithm in the present invention slightly reduces the inference speed, it greatly improves the segmentation accuracy, and its MIoU can reach up to 80.4%, and the inference speed is 16ms / frame. Therefore, it has better comprehensive performance, higher cost performance, and better practical application value.

[0192] Example 2

[0193] In this embodiment, the laboratory corridor scene data set experiment is mainly carried out, and the laboratory corridor scene data set is a data set established by a self-set scene. The important purpose of this embodiment is to verify the training effect of the enhanced training module designed by the present invention. When the MCDWA_Net model is trained with the laboratory corridor scene data set, the training batch Batch_size is set to 4, the number of categories is Num_classes is 8, the number of iterations is Epoches is 500, and the learning strategy adopts the stochastic gradient descent (SGD) algorithm, and its momentum is set to 0.9, the initial learning rate is set to 0.05, and the weight decay rate is 0.0001.

[0194] The CEDice loss function of the enhanced training module proposed in the present invention is compared with the training results of using the CrossEntropy loss function and the Dice loss function alone to verify the superiority of the algorithm of the present invention. The change process of the loss value during the training process is as follows: Fig.14 As shown in Figure 2, the MIoU value changes as follows: Fig.15 In addition, Table 4 shows the training effects of various loss functions.

[0195] Table 4 Comparison of training effects of three loss functions

[0196]

[0197] Combination Fig.14As can be seen from Table 4, the CEDice loss function designed by the present invention can reduce the loss value by more than 90% after 90 cycles of training, while the Cross Entropy loss function needs 139 cycles to reduce the loss value by 90% during the training process, and the Dice loss function needs 141 cycles to reduce the loss value by 90% during the training process. It can be seen that the loss function designed by the present invention can better reduce the loss value during training.

[0198] Combination Fig.15 As can be seen from Table 4, in the training process of the laboratory corridor scene dataset, the enhanced training module in the present invention uses the CEDice loss loss function to make the model MIoU as high as 95%, and only needs 9 cycles of training to improve the model's MIoU by 90%, while the Cross Entropy loss function and the Dice loss loss function can make the model accuracy as high as 94.2% and 82%, respectively. They need 22 and 298 cycles of training to improve the model's MIoU by 90%. It can be seen that the training algorithm designed by the present invention can improve the semantic segmentation network model to a higher accuracy more quickly, which is stronger than the separate application of other loss functions.

[0199] At the same time, the present invention also trains the corresponding models of BiSeNet, BiSeNetV2, DFANet, Deeplab V3 and ShuffleNet V2 algorithms on the laboratory corridor scene dataset, and compares the performance of the algorithm of the present invention. The performance of the six algorithms on the laboratory corridor scene dataset is shown in Table 5. The segmentation effects of the six algorithms on the laboratory corridor scene are shown in Table 5. Fig.16 shown.

[0200] Table 5 Performance comparison of MCDWA_Net algorithm and advanced algorithms on the laboratory corridor scene dataset

[0201]

[0202] Table 5 shows that although the inference speed of the MCDWA_Net algorithm proposed in the present invention is relatively slow, its segmentation accuracy is greatly improved. The specific segmentation effects of these algorithms are as follows: Fig.16 As shown, the image segmentation effect of the algorithm proposed in the present invention is obviously better than that of other algorithms.

[0203] In addition, the proposed algorithm has a significant improvement in segmentation accuracy, which is much higher than the other five algorithms, and its computational complexity is relatively low and its reasoning speed is relatively fast. Fig.15It can be seen (the left picture is a relatively simple scene, and the right picture is a relatively complex scene) that the superiority of the algorithm of the present invention is not very obvious in the simple scene of the left picture, but in the complex scene of the right picture, it can be clearly seen that the semantic segmentation of the scene by the other five algorithms is defective, because they cannot completely extract the global semantic information of the image, resulting in relatively poor segmentation effect. Therefore, in the experiment of the laboratory corridor scene, the algorithm proposed by the present invention has better comprehensive performance, higher segmentation accuracy, and faster running speed, which further demonstrates the superiority of the algorithm of the present invention in actual scenes.

[0204] Finally, it should be noted that although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0205] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0206] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0207] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. An image semantic segmentation algorithm based on multi-channel deep weighted aggregation, characterized in that: The following steps are involved: S1, through the low-level semantic channel, auxiliary semantic channel and high-level semantic channel, respectively extract the semantic features with clear category information in the image, the transitional semantic features between low-level semantics and high-level semantics, and the semantic features of the contextual logical relationship in the image; A shallow convolutional structure network is used to construct a low-level semantic channel to extract low-level semantic information; A deep separable convolutional structure network is used to construct an auxiliary semantic channel, and the transitional semantic information obtained by the auxiliary semantic channel is fed back to the high-level semantic channel; A deep convolutional structure network is used to construct a high-level semantic channel to extract high-level semantic information; S2, fusing the three different semantic features obtained in step S1 through weighted aggregation to obtain the global semantic information of the image; S3, the semantic features output by each semantic channel in step S1 and the global semantic information in step S2 are used together to calculate the loss function for training.

2. The image semantic segmentation algorithm based on multi-channel depth weighted aggregation according to claim 1 is characterized in that: The process of extracting low-level semantic information from a shallow convolutional network includes: LS(I H*W )=S3(S2(S1(I H*W ))) Among them, LS(x) is the low-level semantic information extraction process, I H*W is the input image matrix and S is the convolution step size.

3. The image semantic segmentation algorithm based on multi-channel depth weighted aggregation according to claim 1 is characterized in that: The separable convolutional structure network consists of three auxiliary modules connected in series and an attention mechanism module at the tail. Each module transfers the extracted auxiliary semantic information at different levels to the high-level semantic channel. The process of extracting transition semantic information through the auxiliary semantic channel includes: AS(K m*n )=Atten(Aux(Aux(Aux(K m*n )))) Among them, AS(x) is the transition semantic information extraction process, Km*n is the feature matrix of the input auxiliary semantic channel, Aux(x) is the auxiliary module, and Atten(x) is the attention mechanism module; The auxiliary semantic information output by each stage of the auxiliary semantic channel includes: Aux1(K m*n )=Aux(K m*n ) Aux2(K m*n )=Aux(Aux1(K m*n )) Aux3(K m*n )=Aux(Aux2(K m*n )) Aux1'(K m*n )=Up(AS(K m*n ),4) Among them, Up(x,k) is the upsampling of the attention mechanism module to the high-level semantic channel, x is the input, k is the upsampling multiple, and the selected value is 4.

4. The image semantic segmentation algorithm based on multi-channel depth weighted aggregation according to claim 3 is characterized in that: The high-level semantic channel includes a startup module, a feature aggregation module, and a semantic nesting module. The process of extracting high-level semantic information includes: The deep semantic information of the image is extracted by fusing the auxiliary semantic information provided by the starting module and the feature aggregation module with the auxiliary semantic channel. Then, the semantic nesting module is used to integrate the deep semantic information of the auxiliary semantic channel with the high-level semantic channel to obtain the high-level semantic information.

5. The image semantic segmentation algorithm based on multi-channel depth weighted aggregation according to claim 4 is characterized in that: The startup module includes two different downsampling methods: convolution and maximum pooling, and the two output results are connected and then convolved; The feature aggregation module includes two feature aggregation submodules, and uses deep convolution to deepen the number of feature layers. The aggregation process of the feature aggregation module includes: FGB(K' m*n ,s1,s'1)=FG(FG(K' m*n ,s1),s'1) Among them, FGB(x) is the reasoning process of the feature aggregation module, FG(x,s1) represents the reasoning process of the feature aggregation submodule, and K' m*n is the input feature matrix, step size s1=2, s'1=1; The reasoning process of the feature aggregation module of the three auxiliary modules in the auxiliary semantic channel includes: FGB1(K m*n )=FGB(ST(K m*n )+Aux1(K m*n )+Aux1'(K m*n ),s1,s'1) FGB2(K m*n )=FGB(FGB1(K m*n )+Aux2(K m*n ),s1,s'1) <h2 style=";text-align:left;direction:ltr">FGB3(K<h2 style=";text-align:left;direction:ltr"> m*n <h2 style=";text-align:left;direction:ltr"> )=FGB(FGB2(K<h2 style=";text-align:left;direction:ltr"> m*n <h2 style=";text-align:left;direction:ltr"> )+Aux3(K<h2 style=";text-align:left;direction:ltr"> m*n <h2 style=";text-align:left;direction:ltr"> ),s1,s'1) The semantic embedding module adopts global average pooling and skip connection structure to deeply fuse the auxiliary semantic information provided by the high-level semantic channel and the auxiliary semantic channel; The process of extracting semantic information from high-level semantic channels includes: AS(K m*n )=SEB(FGB3(K m*n )) Among them, AS(x) is the semantic information extraction process of the high-level semantic channel, and SEB(x) is the reasoning process of the semantic nested module.

6. The image semantic segmentation algorithm based on multi-channel depth weighted aggregation according to claim 1, characterized in that: The fusion of semantic features in step S2 includes weighted aggregation and fusion of three types of semantic features. Weighted aggregation includes: updating the low-level semantic channel, auxiliary semantic channel and high-level semantic channel to obtain the weighted weights ε1, ε2 and ε3 of the semantic features respectively. The weight update is adaptively determined by the MIoU = {MIoU1, MIoU2, MIoU3} measured by the three types of semantic features in the validation set. The network weights of each channel are used to obtain MIoU on the validation set, and then the MIoU value is used to update the weighted weights of the three types of semantic features according to the following formula, including: The weighting process of three types of semantic features includes: Wherein, Up is an upsampling with a magnification of 2, Up(x) represents the upsampling process, Conv is a convolution operation, and the calculation of sigmoid(x) includes:

7. The image semantic segmentation algorithm based on multi-channel depth weighted aggregation according to claim 6 is characterized in that: The fusion of the three types of semantic features includes multiplying the weighted results obtained from the two semantic channels and then performing a 3*3 convolution with a step size of 1. The pairwise aggregation process of the three types of semantic features includes: Among them, SA1(x), SA2(x) and SA3(x) are the pairwise aggregation processes of three types of semantic features respectively; The compound process of the three-way semantic features obtained by the three types of semantic channels includes: MSA(x)=SA1(x)+SA2(x)+SA3(x) Among them, MSA(x) is a composite process of three-way semantic features.

8. The image semantic segmentation algorithm based on multi-channel depth weighted aggregation according to claim 1, characterized in that: In step S3, the semantic features output by each channel are upsampled and the loss is calculated together with the global semantic information of the image to perform stochastic gradient descent learning. The loss function includes the calculation of weights, the calculation of loss functions, and the determination of loss functions in the comprehensive training mechanism. The calculation of weights includes: Among them, α is the weight, n is the number of categories; The calculation of the loss function includes: CED(x,y)=CEL(x,y)+α*DL(x,y) Among them, x is the predicted data, y is the real data, CEL(x,y) is the Cross-Entropy loss function, and DL(x,y) is the Dice loss function; The determination of the loss function in the comprehensive training mechanism includes: Among them, T is the true label, P re Output labels for global semantic information. A1, A2, and A3 are the output labels of low-level semantic channel, auxiliary semantic channel, and high-level semantic channel, respectively.

9. An image semantic segmentation system based on multi-channel depth weighted aggregation, comprising the image semantic segmentation algorithm based on multi-channel depth weighted aggregation according to claim 5, characterized in that: include: Three-channel semantic representation model, three types of semantic weighted aggregation modules and enhanced training module; The three-channel semantic representation model includes a low-level semantic channel, an auxiliary semantic channel and a high-level semantic channel; the auxiliary semantic channel includes an auxiliary module and an attention mechanism module, and the high-level semantic channel includes a startup module, a feature aggregation module and a semantic nesting module.

Images