A semantic segmentation method and device based on graph structure neural network

Abstract

The present invention discloses a semantic segmentation method and device based on a graph structure neural network. A class semantic enhancement module (CSE) is proposed, which uses a graph model to create a graph structure between channels, and further outputs a "channel"-"object" relationship matrix to reconstruct a feature map. In addition, a fully convolutional network layer that integrates object prior information is created based on the class semantic enhancement module, and a fine segmentation result is generated through coarse segmentation and feature maps. Based on the above two modules, a class semantic enhancement network (CSENet) is proposed, which captures the interdependence between channels and the "channel"-"object" relationship. A coarse-to-fine segmentation strategy is used, and the above CSENet and CPFC layers are stacked in sequence, so as to achieve gradual optimization of the coarse segmentation results, and finally output the fine segmentation results as the final network output. Experiments have shown that this method can effectively improve the performance of existing semantic segmentation networks.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, semantic segmentation, and in particular to a semantic segmentation method and device based on a graph structure neural network. Background Art

[0002] Semantic segmentation is a challenging and fundamental task in computer vision, which aims to understand and segment scenes. In the real world, objects in a scene are not independent, but interact with each other to form complex scenes. Accurately capturing the interdependencies between objects helps to understand the scene semantics and thus complete pixel-level segmentation in the scene.

[0003] Since the fully convolutional network [1] Since the emergence of Fully convolutional networks (FCN), FCN-based methods have been the main solution. Several recent works using multi-scale strategies have successfully utilized object context information. DeepLab series of methods [2][3][4] We continue to explore convolutional modules with different dilation rates to expand the receptive field and enhance object context features. [5] Use global average pooling to learn global context. As an extension and extension of the above methods, some work [6][7][8][9]

[10] By aggregating feature maps at multiple scales, a wider range of semantic information is obtained. Although the multi-scale method broadens the receptive field, its method causes some local information to be lost and the correlation between objects is ignored.

[0004] Methods based on "relationships" have performed well in recent years because they are not limited by the range of the receptive field. According to the adopted relationship dimension, they can be divided into two groups. One is pixel relationship, which is used to study the interaction between pixels. The other is region relationship, which aims to study the characteristics of a specific region composed of certain pixels. Adaptive region, prior spatial distribution of objects, and each feature map of a channel are commonly used definitions of regions. For methods based on pixel relationships, DANet

[11] The correlation between pixels is studied and coherent feature representations of similar pixels are given. ACFNet

[12] We further investigate the higher-order relationships between pixels in order to track the interdependencies between objects.

[13] A factorized pixel-level attention is proposed to reduce the computational cost, but establishing correlations between pixels still significantly increases the complexity of the model.

[0005] Although region-relation-based methods can reduce model complexity, the performance of these methods depends largely on the quality of region segmentation due to the complexity and variability of image scenes and semantics. Adaptive region segmentation methods perform poorly due to the lack of strong discriminative information about the target region. DANet's attention module regards channels as feature representations of objects and enhances the representation of each channel by weighted summation between channels. The implicit, non-replicable object region features of channels still face the problem of poor performance. ACFNet directly uses coarse segmentation as a priori spatial distribution to enhance the features of each pixel. However, each channel describes the object from various aspects (locally or globally) and is not limited to the spatial distribution of the object, which leads to inaccurate feature optimization.

[0006] [1].Long J, Shelhamer E, Darrell T.Fully convolutional networks forsemantic segmentation[C] / / Proceedings of the IEEE conference on computervision and pattern recognition.2015: 3431-3440.

[0007] [2].Chen LC, Papandreou G, Kokkinos L, et al. Deeplab: Semantic imagesegmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE transactions on pattern analysis and machineintelligence, 2017, 40(4): 834-848.

[0008] [3].Chen LC, Zhu Y, Papandreou G, et al.Encoder-decoder with atrousseparable convolution for semantic image segmentation[C] / / Proceedings of theEuropean conference on computer vision(ECCV).2018: 801-818.

[0009] [4].Chen L C,Papandreou G,Schroff F,et al.Rethinking atrousconvolution for semantic image segmentation[J]arXiv preprint arXiv:1706.05587,2017.

[0010] [5].Zhao H,Shi J,Qi X,et al.Pyramid scene parsing network[C] / / Proceedings of the lEEE conference on computer vision and patternrecognition.2017:2881-2890.

[0011] [6].Zhang H,Dana K,Shi J,et al.Context encoding for semanticsegmentation[C] / / Proceedings of the IEEE conference on Computer Vision andPattern Recognition.2018:7151-7160.

[0012] [7].Yang M,Yu K,Zhang C,et al.Denseaspp for sementic segmentation instreet scenes[C] / / Proceedings of the lEEE conference on computer Vision andpattern recognition.2018:3684-3692.

[0013] [8].He J,Deng Z,Zhou L,et al.Adaptive pyramid context network forsemantic segmentation[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision end Pattern Recognition.2019:7519-7528.

[0014] [9]. Lin D, Shen D, Shen S, et al. Zigzagnet: Fusing top-down and bottom-up context for object segmentation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 7490-7499.

[0015]

[10] . Fu J, Liu J, Tian H, et al. Dual attention network for scene segmentation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 3146-3154.

[0016]

[11] . Zhang H, Zhang H, Wang C, et al. Co-occurrent features in semantic segmentation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 548-557.

[0017]

[12] . Huang L, Yuan Y, Guo J, et al. Interlaced sparse self-attention for semantic segmentation[J]. arXiv preprint arXiv:1907.12273, 2019.

[0018]

[13] . He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778. Summary of the Invention

[0019] The main application scenario of this invention is to solve the problem of semantic segmentation, that is, to assign the correct category label to each pixel in the input image. Semantic segmentation has a very wide range of application scenarios in the industry, involving medical image segmentation, autonomous driving, facial recognition, geological detection, etc.

[0020] The purpose of the present invention is to address the deficiencies of the prior art and to solve the above-mentioned problems of inaccurate feature representation and high complexity of the existing methods. The present invention provides a semantic segmentation method and device based on a graph structured neural network.

[0021] First, the class semantic enhancement module (CSE) is constructed using the graph attention mechanism. This module creates a graph structure between channels and further outputs a more accurate "channel"-"object" similarity matrix to reconstruct a distinguishable class semantic representation. Secondly, based on the class semantic enhancement module, a fully convolutional network layer (CPFC) containing object prior information is constructed. This layer generates a more refined segmentation result through the coarse segmentation result and the reconstructed feature map. Based on the above two modules, the class semantic enhancement network (CSENet) is formed, which can capture the interdependence between different dimensions of the feature map and the prior information between "channel" and "object", thereby generating the final segmentation result.

[0022] The purpose of the present invention can be achieved by the following technical method: a semantic segmentation method based on a neural network with a graph structure, the method comprising the following steps:

[0023] (1) Obtain the image to be segmented and input it into the residual network for feature extraction; and then pass the extracted feature map through the convolutional network to generate a preliminary segmentation result;

[0024] (2) Inputting the preliminary segmentation result into a number of sequentially connected class semantic enhancement networks to gradually refine the segmentation result, wherein the class semantic enhancement network includes a class semantic enhancement module and a fully convolutional network layer that integrates object prior information; the class semantic enhancement module takes the feature map and segmentation result generated by the previous class semantic enhancement network as input, outputs a new feature map and a joint probability density matrix to the fully convolutional network layer that integrates object prior information, and then outputs the segmentation result;

[0025] (3) The segmentation result after the last class semantic enhancement network refinement is restored to the original resolution of the input image, and the category with the highest confidence is taken as the final category of each pixel to obtain the semantically segmented image.

[0026] Furthermore, the semantic enhancement module in the semantic enhancement network converts the feature map and segmentation result of the previous semantic enhancement network into matrices and multiplies them to obtain an object-channel relationship matrix, linearly maps the relationship matrix using k learnable parameters, and calculates the cosine similarity of each dimensional vector after mapping to form k adjacency matrices A k (k=1,2,3), A k Element A in i，j The value of can characterize the degree of correlation between feature map channels i and j. The larger its value, the closer the correlation between the two channels. The graph neural network is used to interact and aggregate class semantic information on closely related dimensions to generate a joint probability density matrix. The relationship matrix and the joint probability density matrix are multiplied element by element and then multiplied with the matrix converted from the segmentation result to obtain the reconstructed feature map, which is fused with the feature map generated by the previous class semantic enhancement network to output the fused feature map.

[0027] Furthermore, the specific process of generating the joint probability density matrix is ​​as follows: defined as P∈R C*N , where R C*N It is a C*N dimensional real vector space, and its mathematical expression is as follows:

[0028] P=σ(||A k Rψ k )

[0029] where ψ k ∈R N*N is the learnable parameter matrix of the kth graph structure; σ represents the sigmoid function; the element P in the matrix P ij Represents the feature map F on the i-th dimension of the feature map i The probability of an object with label number j in .

[0030] Furthermore, the fully convolutional network layer that integrates the object prior information specifically processes the feature map and the joint probability density matrix as follows:

[0031]

[0032] in is the fine segmentation result, C is the number of feature map channels, p ji is the element in the jth row and ith column of the above joint probability density matrix P, w ij is the element in the i-th row and j-th column of the learnable convolution kernel W, f j is the i-th channel of the feature map, b i is the i-th learnable bias term.

[0033] Furthermore, the semantic enhancement network uses the cross entropy function to calculate the losses of the semantic enhancement module and the fully convolutional network layer that integrates the object prior information respectively. The final loss is obtained by weighted summing the above two losses, and the proportion of the two losses in the final loss is controlled by the exponential decay factor.

[0034] Furthermore, the process of restoring the refined segmentation results to the original resolution of the input image is achieved through bilinear interpolation.

[0035] In a second aspect, the present invention further provides a device, comprising:

[0036] one or more processors;

[0037] A memory for storing one or more programs;

[0038] When the one or more programs are executed by the one or more processors, the one or more processors execute the semantic segmentation method based on the graph structure neural network.

[0039] In a third aspect, the present invention also provides a computer-readable storage medium for storing one or more computer programs, wherein the one or more computer programs include program codes. When the computer programs are run on a computer, the program codes are used to execute the above-mentioned semantic segmentation method based on a graph structured neural network.

[0040] Beneficial effects of the present invention:

[0041] (1) A graph-based semantic enhancement module is proposed. This module models the channel-object correlation and the channel-channel correlation. By organically combining the two correlations, the module learns the category semantic information that takes into account both intra-class consistency and inter-class distinguishability.

[0042] (2) A fully convolutional network layer that integrates object prior information is proposed. This layer applies the object prior information to the convolution kernel parameters. This operation can effectively improve the accuracy of the segmentation results.

[0043] (3) A coarse-to-fine segmentation strategy is used to stack the above CSENet and CPFC layers in sequence, thereby achieving a gradual optimization of the coarse segmentation results, and finally outputting the fine segmentation results as the final network output. Experiments have shown that this method can effectively improve the performance of existing semantic segmentation networks.

[0044] (4) The proposed network has few parameters and can be applied to most segmentation methods based on deep neural networks, achieving significant performance improvements. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1This is the overall framework structure of the deep neural network of the present invention.

[0046] Figure 2 Schematic diagram of the Class Semantic Enhancement Module (CSE) of the present invention.

[0047] Figure 3 Schematic diagram of the fully convolutional network layer (CPFC) that integrates object prior information in the present invention.

[0048] Figure 4 This is an example of an image to be segmented that is input into the present invention.

[0049] Figure 5 This is an example of a rough segmentation result generated by the present invention.

[0050] Figure 6 This is an example of the fine segmentation result finally output by the present invention.

[0051] Figure 7 It is a structural diagram of the semantic segmentation device based on the graph structure neural network of the present invention. DETAILED DESCRIPTION

[0052] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0053] The present invention provides a neural network for semantic segmentation tasks based on a graph structure, which specifically includes the following steps:

[0054] 1. Problem description and variable definition

[0055] For the semantic segmentation problem with a total number of categories N, the goal is to assign the correct category label to each pixel of the input image. The existing standard deep neural network-based method is: for a given input image I∈R 3*H*W , where H and W are the image height and width respectively. First, the image I is input into the backbone network (such as the residual network) to generate the feature map F∈R C*h*w , where C is the number of feature map channels, h, w are the height and width values ​​after downsampling. Then, the convolution kernel ω∈R is used N *C*1*1 Convolution operation is performed with the feature map to generate X0∈R N*h*w Finally, the interpolation function and argmax function act on X0 respectively, and the output segmentation result Y0∈R H*W .

[0056] Attach Figure 4Take the image to be segmented as an example, which contains multiple categories of "targets", such as pedestrians, roads, signs, etc. The goal of the semantic segmentation task is to assign correct labels to each category of "target" in the image. The present invention aims to further process the inaccurate segmentation results (i.e., "coarse segmentation" results) to obtain more accurate segmentation results (i.e., "fine segmentation" results). Figure 5 As an example of the rough segmentation result, it can be observed that the neural network does not accurately segment the "sidewalk" in the dotted box in the lower left corner of the image, and misclassifies the "signboard" in the solid box in the upper middle part of the image. Figure 6 The "fine segmentation" result output by the present invention shows that the segmentation of the "sidewalk" in the lower left corner is more accurate, and the misclassification of the "sign" is eliminated, which enhances the accuracy of the neural network in the segmentation task to a certain extent.

[0057] As attached Figure 1 As shown, the present invention proposes a coarse-to-fine segmentation network, called Class Semantic Enhancement Network (CSENet), which can be flexibly inserted behind the above standard segmentation model to generate more accurate results and gradually refine the feature map F and the coarse segmentation result X0.

[0058] Specifically, Figure 2 As shown in the figure, the class semantic enhancement network proposed in the present invention is composed of a class semantic enhancement module (CSE) fused with a fully convolutional network layer (CPFC) of object prior information, and the segmentation result is gradually refined by stacking n class semantic enhancement networks in sequence. Among them, the CSE module reconstructs the feature map using the high-order information between channels; the CPFC module combines the class-channel relationship map with the reconstructed feature map to obtain a fine-grained segmentation result {X i ∈R N*HW , i = 1, 2, 3...n}, where X i Represents the segmentation result of the i-th unit.

[0059] 2. Feature extraction network

[0060] Most semantic segmentation methods based on deep neural networks first input the image to be segmented into the backbone network, extract image features, and output high-dimensional feature maps with semantic information for subsequent processing. The present invention uses the more commonly used and advanced residual network (Residual Networks, ResNet) [He K, Zhang X, Ren S, et al. Deep learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.] as the feature extraction network (i.e., the backbone network). The residual network belongs to a convolutional neural network, which is mainly composed of a convolutional layer and a nonlinear layer. The network introduces a "residual" structure into the classic convolutional neural network, which solves the problems of gradient disappearance, gradient explosion, network degradation, etc. to a certain extent, and improves the performance of traditional convolutional neural networks. Its residual structure can be described by the following mathematical expression:

[0061] H(X)=F(X)+X

[0062] Among them, X is the output of the upper network; F represents the mapping function of this layer of network for the input; and H is the final output mapping of this layer of network.

[0063] From the above formula, we can see that the output of each layer of the residual network is the sum of the output after network parameter mapping and the original input, that is, the network parameters only need to fit the "residual" between the "input" and the "ideal output".

[0064] 3. Overall structure description

[0065] The present invention first introduces the overall structure of the proposed semantic enhancement network and explains its relationship with other modules such as the backbone network. Figure 1 As shown, the overall process of the graph-based neural network for semantic segmentation proposed in the present invention can be described by the following mathematical formula:

[0066] F0=Backbone(I)

[0067] X0=FC(F0)

[0068] F i , X i =CSENet i (F i-1 , X i-1 )i=1,2,3…n

[0069] Y n =Argmax(Bilinear(Xn ))

[0070] Among them, I is the input image to be segmented; Backbone represents the backbone network mentioned above; F0 is the output feature map of the backbone network; FC is the convolutional network layer used by the FCN network to generate the segmentation result X0; CSENet i is the semantic enhancement network of the i-th class; F i and X i is the feature map and segmentation result output by the i-th CSENet; X n is the segmentation result output after the nth iteration; Bilinear represents the bilinear interpolation operation; Argmax is the maximum value operation on the category dimension; Y n It is the final segmentation result output by the network.

[0071] From the above formula, we can see that the proposed semantic information enhancement network takes feature maps and rough segmentation results as input, and outputs feature maps with clearer semantic information and more refined segmentation results. Specifically, the network first uses the convolutional layers in the backbone network and FCN network to obtain the original feature map F0 and the preliminary segmentation result X0. Then, the two are input into a temporal network composed of n CSENets to output the segmentation result X0 after n iterations. n Finally, the result is restored to the original resolution of the input image through bilinear interpolation, and the category with the highest confidence is taken as the final category of each pixel. The output is recorded as Y n .

[0072] The semantic-like enhancement network proposed in the present invention consists of two modules, namely the semantic-like enhancement module and the fully convolutional network layer that integrates object prior information. The input and output of the two modules can be described by the following formula:

[0073] F i , P i =CSE i (F i-1 , X i-1 )

[0074] X i =CPFC i (F i , P i )

[0075] Among them, CSE i and CPFC i They are respectively the semantic enhancement module of the i-th class and the fully convolutional network layer that integrates the object prior information.

[0076] From the above formula, we can see that the CSE module takes the feature map generated by the previous round (i.e., the i-1th round) and the coarse segmentation result as input, and outputs a new feature map Fi At the same time, output the joint probability density matrix P i ; CPFC layer is F i , Px is used as input, and the segmentation result X after the i-th iteration is output i .

[0077] 4. CSE module

[0078] (1) Calculate the correlation between categories and feature maps

[0079] The present invention defines the correlation between classes and feature map channels as follows:

[0080]

[0081] Among them, F i represents the feature map on the i-th dimension, X j represents the jth rough segmentation graph; T represents the transposition operation; represents the sum of each element in the j-th rough segmentation map; R i，j Indicates F i The average response value of the jth class. The higher the R i，j This indicates that the i-th dimension and the j-th class are more semantically related, and vice versa.

[0082] (2) Learning the joint probability density matrix P

[0083] First, the present invention establishes K independent graph structures to represent the interdependence between different dimensions of features. Their adjacency matrix is ​​defined as follows:

[0084]

[0085] in represents the inner product operation, is a learnable projection function, R i A vector representing the strength of the relationship between the i-th dimension of the feature map and the objects corresponding to the various label numbers. is the edge weight between the i-th and j-th feature dimensions. The higher Indicates F i and F j More likely to contain the same class.

[0086] Secondly, the present invention uses a standard multi-head graph model to interact and aggregate class semantics on interdependent dimensions, and finally generates a joint probability density matrix, defined as P∈R C*N , its mathematical expression is as follows:

[0087] P=σ(IIA k Rψ k )

[0088] Among them A k is the kth adjacency matrix between feature map channels, R is the channel-object relationship matrix, ψ k ∈R N*N is the learnable parameter matrix of the kth graph structure; σ represents the sigmoid function; the i-th and j-th elements P in the matrix P ij Indicates F i The probability of involving the jth class.

[0089] (3) Feature reconstruction

[0090] Use the joint probability density matrix P to adjust R and further reconstruct the feature map

[0091]

[0092] Then, a convolution kernel with a dimension of 2C*C*1*1 is used to combine the feature map F generated by the residual network with Fusion, output new feature map F', where ⊙ represents the corresponding element multiplication operation.

[0093] 5. CPFC layer

[0094] As attached Figure 3 As shown in the figure, the CPFC layer uses the joint probability density matrix P output by the CSE module and the reconstructed new feature map F′ as input, uses the joint probability density matrix P to multiply the learnable convolution kernel element by element, and then performs a convolution operation with the feature map F′ to output the refined segmentation result. The mathematical expression of this process is as follows:

[0095]

[0096] where p ji is the element in the jth row and ith column of the above joint probability density matrix P, w ij is the element in the i-th row and j-th column of the learnable convolution kernel W, f j is the i-th channel of feature map F, b i is the i-th learnable bias term.

[0097] 6. Loss Function

[0098] In the design of the loss function, the present invention uses the cross entropy function to calculate the loss L of the rough segmentation and the fine segmentation respectively. c and L f The mathematical expression of the cross entropy function is:

[0099]

[0100] Among them, x is the network input, p(x) is the expected output, q(x) is the actual output of the network, and H(p, q) is the cross entropy.

[0101] The final loss L can be described by the following formula:

[0102] L=(a+γ)L c +(b-γ+0.1)L f

[0103]

[0104] Among them, L c , L f are the coarse segmentation loss and fine segmentation loss calculated using the cross entropy loss above; L is the loss function of the network’s final application; a and b are constants, a = 0.4, b = 0.9; iter is the current iteration number of the class semantic enhancement network (i.e., the current class semantic enhancement network); γ is an exponential decay factor, whose value decreases when the iteration number is less than the threshold iteration number iter γ This design allows the network to pay more attention to the coarse segmentation results in the early stage of training and pay more attention to the fine segmentation results in the later stage of the network.

[0105] 7. Output segmentation results

[0106] After n rounds of iterations, we get X n After that, bilinear interpolation and maximum value operation on the category dimension are performed to obtain the final segmentation result Y n .

[0107] Given points (x1, y1), (x1, y2), (x2, y1), (x2, y2), and their values ​​q 11 ,q 12 ,q 21 ,q 22 , the value q of the point (x, y) to be found is given by the following formula:

[0108]

[0109] According to this law, the present invention performs X with dimension N*hw on n , that is, the low-resolution segmentation result, is upsampled to obtain a matrix of dimension N*HW, where H and W are the image height and width respectively, N is the total number of categories, and l represents the lth category. Finally, the N*1-dimensional vector corresponding to the pixel in the i-th row and j-th column of the output after the n-th iteration is Take the maximum value As the final category of the pixel, that is:

[0110]

[0111]

[0112] Income Y n The dimensions are H*W, The value of represents the category label of the pixel in the i-th row and j-th column.

[0113] The embodiments of the present invention for autonomous driving tasks are as follows:

[0114] (1) Preparation

[0115] First, we need to prepare the Cityscapes dataset required for the experiment. This dataset contains 5,000 images of driving scenes in urban environments. Correct segmentation of this dataset, that is, successfully distinguishing pedestrians, roads, signs, etc., can be effectively applied in autonomous driving tasks.

[0116] Secondly, load the pre-trained parameters of the backbone network (the download link is https: / / download.pytorch.org / models / resnet50-19c8e357.pth).

[0117] (2) Set hyperparameters, mainly including the following hyperparameters:

[0118] Hyperparameter Name Initial learning rate Void rate a b epoch Batch size n value 0.009 8 0.4 0.9 120 16 3

[0119] (3) Select any data set to train the network, and test the network accuracy after training. In the experiment with the backbone network being ResNet-50 and the data set being Cityscapes, the accuracy of the original FCN network was 72.25%. After using the CSENet proposed in the present invention, the accuracy on the Cityscapes data set was 75.18%.

[0120] Corresponding to the aforementioned embodiment of the semantic segmentation method based on a graph structure neural network, the present invention also provides an embodiment of a semantic segmentation device based on a graph structure neural network.

[0121] See also Figure 7 A semantic segmentation device based on a graph structure neural network provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the semantic segmentation method based on a graph structure neural network in the above embodiment.

[0122] The embodiments of the semantic segmentation device based on the graph structure neural network of the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 7 As shown, it is a hardware structure diagram of any device with data processing capability where the semantic segmentation device based on graph structure neural network of the present invention is located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0123] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0124] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0125] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the semantic segmentation method based on a graph structure neural network in the above embodiment is implemented.

[0126] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0127] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A semantic segmentation method based on a graph structure neural network, characterized in that: The method comprises the following steps: (1) Obtain the image to be segmented and input it into the residual network for feature extraction; and then pass the extracted feature map through the convolutional network to generate a preliminary segmentation result; (2) The preliminary segmentation result is input into several sequentially connected class semantic enhancement networks to gradually refine the segmentation result. The class semantic enhancement network includes a class semantic enhancement module and a full convolutional network layer that integrates object prior information; the class semantic enhancement module takes the feature map and segmentation result generated by the previous class semantic enhancement network as input, outputs the new feature map and the joint probability density matrix to the full convolutional network layer that integrates object prior information, and then outputs the segmentation result; specifically: the class semantic enhancement module in the class semantic enhancement network converts the feature map and segmentation result of the previous class semantic enhancement network into a matrix and then multiplies them to obtain an object-channel relationship matrix, uses k learnable parameters to linearly map the relationship matrix, and calculates the cosine similarity of each dimensional vector after mapping to form k adjacency matrices A k (k=1,2,3), A k Element A in i，j The value of can represent the degree of correlation between feature map channels i and j. The larger its value, the closer the correlation between the two channels. The graph neural network is used to interact and aggregate class semantic information on closely related dimensions to generate a joint probability density matrix. The relationship matrix and the joint probability density matrix are multiplied element by element and then multiplied with the matrix converted from the segmentation result to obtain the reconstructed feature map, which is fused with the feature map generated by the previous class semantic enhancement network to output the fused feature map. The specific process of generating the joint probability density matrix is ​​as follows: It is defined as P∈R C*N , where R C*N It is a C*N dimensional real vector space, and its mathematical expression is as follows: P=σ(||A k Rψ k ) where ψ k ∈R N*N is the learnable parameter matrix of the kth graph structure; σ represents the sigmoid function; the element P in the matrix P ij Represents the feature map F on the i-th dimension of the feature map i The probability of an object with label number j in ; The specific processing process of the fully convolutional network layer that integrates the object prior information on the feature map and the joint probability density matrix is ​​as follows: in is the fine segmentation result, C is the number of feature map channels, pji is the element in the jth row and ith column of the above joint probability density matrix P, and w ij is the element in the i-th row and j-th column of the learnable convolution kernel W, f j is the i-th channel of the feature map, b i is the i-th learnable bias term; (3) The segmentation result after the last class semantic enhancement network refinement is restored to the original resolution of the input image, and the category with the highest confidence is taken as the final category of each pixel to obtain the semantically segmented image.

2. According to claim 1, a semantic segmentation method based on a graph structure neural network is characterized in that: The semantic enhancement network uses the cross entropy function to calculate the losses of the semantic enhancement module and the fully convolutional network layer that integrates the object prior information respectively. The final loss is obtained by weighted summing the above two losses, and the proportion of the two losses in the final loss is controlled by the exponential decay factor.

3. The semantic segmentation method based on graph structure neural network according to claim 1, characterized in that: The process of restoring the refined segmentation results to the original resolution of the input image is achieved through bilinear interpolation.

4. A device, characterized in that: The device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are caused to perform the method according to any one of claims 1 to 3.

5. A computer-readable storage medium for storing one or more computer programs, wherein the one or more computer programs include program codes. When the computer programs are run on a computer, the program codes are used to execute the semantic segmentation method based on a graph structured neural network as described in any one of claims 1 to 3.

Images