A foggy traffic scene image semantic segmentation method and related equipment

Abstract

The present application provides a kind of foggy traffic scene image semantic segmentation method and related equipment, comprising: obtaining foggy traffic scene image set;Fuzzy image semantic segmentation network including double-task feature extraction module, mapping module for projecting image from Euclidean space to graph space, confidence pairing enhancement module for capturing the dependency between feature maps, back mapping module for projecting image from graph space to Euclidean space, mutual supervision module is constructed;Foggy traffic scene image set is input fuzzy image semantic segmentation network and is trained, obtains the fuzzy image semantic segmentation network after training;The target traffic scene image to be processed is input fuzzy image semantic segmentation network after training and carries out semantic segmentation, obtains segmentation result;Compared with prior art, it is still possible to improve the accuracy of semantic segmentation according to the guidance of the same task under the condition that the label data is defective, so as to improve the safety of intelligent driving.

Description

Technical Field

[0001] This invention relates to the field of image information processing technology, and in particular to a method and related equipment for semantic segmentation of images of traffic scenes in foggy weather. Background Technology

[0002] In recent years, with the continuous development of deep learning, the field of computer vision has also achieved remarkable results. An increasing number of traffic scenes require accurate and efficient segmentation techniques, making semantic segmentation of traffic scene images a key research area. Semantic segmentation of traffic scene images involves studying the low-level details of traffic scene images and then classifying the pixels of the traffic scene images into corresponding categories.

[0003] In the field of intelligent driving, accurately identifying traffic targets under various driving conditions is one of the important tasks. Fog weather, as one of the most common weather conditions in driving scenarios, brings noise that has a significant impact on optical cameras. This results in a significant inherent similarity between the traffic target to be detected and the traffic background, which poses a considerable safety hazard during intelligent driving.

[0004] Therefore, the existing technology discloses a dual-subnet DSNet object detection method based on multi-task learning to solve the object detection problem in foggy environments; however, this method only uses a small amount of information from the secondary task to directly assist the main task, and then optimizes it through a joint loss function, ignoring the cooperation relationship between tasks and not taking into account the characteristics of specific traffic scenarios. For foggy traffic scene images, it only focuses on the prediction of the internal points of the target and ignores its contour information, resulting in poor generalization ability of the model. Summary of the Invention

[0005] This invention provides a semantic segmentation method and related equipment for traffic scene images in foggy weather, with the aim of improving the accuracy of semantic segmentation when the label data is defective.

[0006] To achieve the above objectives, the present invention provides a semantic segmentation method for foggy traffic scene images, comprising:

[0007] Step 1: Obtain a set of images of traffic scenes in foggy weather;

[0008] Step 2: Construct a fuzzy image semantic segmentation network. The fuzzy image semantic segmentation model includes a dual-task feature extraction module, a mapping module for projecting the image from Euclidean space to graph space, a confidence pairing enhancement module for capturing the dependencies between feature maps, an anti-mapping module for projecting the image from graph space to Euclidean space, and a mutual supervision module, which are connected in sequence.

[0009] Step 3: Input the foggy traffic scene image set into the fuzzy image semantic segmentation network for training to obtain the trained fuzzy image semantic segmentation network;

[0010] Step 4: Input the target traffic scene image to be processed into the trained fuzzy image semantic segmentation network for semantic segmentation to obtain the segmentation result.

[0011] Furthermore, step 3 includes:

[0012] Preprocess the foggy traffic scene image set to obtain the preprocessed foggy traffic scene image set, instance segmentation labels, and contour labels;

[0013] The preprocessed foggy traffic scene image set, instance segmentation labels, and contour labels are input into the fuzzy image semantic segmentation network for training, resulting in the trained fuzzy image semantic segmentation network.

[0014] Furthermore, the dual-task feature extraction module includes:

[0015] First residual block, second residual block, third residual block, fourth residual block, first dilated convolutional layer, second dilated convolutional layer, third dilated convolutional layer, first convolutional layer and splicing layer;

[0016] The input of the first residual block is the input of the fuzzy image semantic segmentation network;

[0017] The output of the first residual block is connected to the input of the second residual block. The output of the second residual block is connected to the input of the third residual block. The output of the third residual block is connected to the input of the fourth residual block. The output of the fourth residual block is connected to the first input of the mapping module, the input of the first dilated convolutional layer, the input of the second dilated convolutional layer, the input of the third dilated convolutional layer, and the input of the first convolutional layer. The outputs of the first dilated convolutional layer, the second dilated convolutional layer, the third dilated convolutional layer, and the first convolutional layer are all connected to the input of the splicing layer. The output of the splicing layer is connected to the second input of the mapping module.

[0018] The preprocessed foggy traffic scene image set is input into the dual-task feature module, and basic features are extracted through the first residual block, the second residual block, the third residual block and the fourth residual block to obtain the first semantic feature map and the first contour feature map.

[0019] The first semantic feature map is input into the first dilated convolutional layer, the second dilated convolutional layer, the third dilated convolutional layer, and the first convolutional layer, respectively, to obtain semantic feature maps of multiple scales;

[0020] The semantic feature maps of multiple scales, the first semantic feature map, and the first contour feature map are input into the concatenation layer and fused to obtain the second semantic feature map that fuses semantic features and contour features.

[0021] The first contour feature map and the second semantic feature map are input into the mapping module for projection to obtain the contour map and semantic map in the graph space.

[0022] Furthermore, the confidence-matching enhancement module includes:

[0023] First inter-graph reasoning module, second inter-graph reasoning module, intra-graph reasoning module;

[0024] The first input terminal of the first inter-graph reasoning module, the first input terminal of the second inter-graph reasoning module, and the second input terminal of the second inter-graph reasoning module are all connected to the first output terminal of the mapping module. The second input terminal and the third input terminal of the first inter-graph reasoning module are all connected to the second output terminal of the mapping module. The output terminal of the first inter-graph reasoning module is connected to the third input terminal of the second inter-graph reasoning module and the second input terminal of the intra-graph reasoning module, respectively. The output terminal of the second inter-graph reasoning module is connected to the first input terminal of the intra-graph reasoning module. The first output terminal of the intra-graph reasoning module is connected to the first input terminal of the anti-mapping module. The second output terminal of the intra-graph reasoning module is connected to the second input terminal of the anti-mapping module.

[0025] The contour map and semantic map are input into the confidence pairing enhancement module. The first inter-graph reasoning module performs cross-reasoning on the semantic map and the contour map to capture the dependency relationship between the semantic map and the contour map, and obtains the enhanced semantic map.

[0026] The second inter-graph reasoning module performs cross-reasoning on the enhanced semantic graph and the contour graph to capture the dependency relationship between the enhanced semantic graph and the contour graph, and obtains the enhanced contour graph.

[0027] The enhanced semantic map and the enhanced contour map are input into the in-graph reasoning module for in-graph feature extraction, resulting in the enhanced semantic feature map and the enhanced contour feature map.

[0028] The enhanced semantic feature map and the enhanced contour feature map are input into the inverse mapping module for inverse projection to obtain the enhanced semantic feature map and the enhanced contour feature map in the original space.

[0029] Furthermore, both the first and second inter-graph reasoning modules include:

[0030] The system consists of a second convolutional layer, a third convolutional layer, a first multilayer perceptron, a second multilayer perceptron, a third multilayer perceptron, a first multiplier, a first adder, a second multiplier, a second adder, and a position compensator.

[0031] The input of the second convolutional layer is connected to the first output of the mapping module and the first input of the second adder, respectively. The output of the second convolutional layer is connected to the input of the first multilayer perceptron. The output of the first multilayer perceptron is connected to the first input of the first multiplier and the first input of the second adder, respectively. The second output of the first multilayer perceptron is connected to the first input of the first multiplier.

[0032] The input of the third convolutional layer is connected to the second output of the mapping module. The output of the third convolutional layer is connected to the inputs of the second and third multilayer perceptrons. The output of the second multilayer perceptron is connected to the second input of the first multiplier. The outputs of the second multiplier and the position compensator are both connected to the input of the first adder. The output of the first adder is connected to the first input of the second multiplier. The output of the third multilayer perceptron is connected to the second input of the second multiplier. The output of the second multiplier is connected to the second input of the second adder. The output of the second adder is connected to the input of the in-graph inference module.

[0033] Furthermore, the in-graph reasoning module includes:

[0034] First graph convolutional network and second graph convolutional network;

[0035] The input of the first graph convolutional network is connected to the output of the second adder in the first graph inference module, and the output of the first graph convolutional network is connected to the second input of the inverse mapping module.

[0036] The input of the second graph convolutional network is connected to the output of the second adder in the second graph inference module, and the output of the second graph convolutional network is connected to the first input of the inverse mapping module.

[0037] Furthermore, the mutual supervision module includes:

[0038] Fourth convolutional layer, fifth convolutional layer, sixth convolutional layer, seventh convolutional layer, third adder and fourth adder;

[0039] The input terminals of the fourth and fifth convolutional layers are both connected to the first output terminal of the inverse mapping module. The output terminal of the fourth convolutional layer is connected to the first input terminal of the third adder and the output terminal of the sixth convolutional layer, respectively. The second input terminal of the third adder is connected to the output terminal of the sixth convolutional layer. The input terminals of the sixth and seventh convolutional layers are both connected to the second output terminal of the inverse mapping module. The output terminal of the seventh convolutional layer is connected to the first input terminal of the fourth adder and the output terminal of the fifth convolutional layer, respectively. The second input terminal of the fourth adder is connected to the output terminal of the fifth convolutional layer.

[0040] The enhanced semantic feature map in the original space, the enhanced contour feature map in the original space, the contour label and the instance segmentation label are all input into the mutual supervision module. The enhanced semantic feature map in the original space is transformed through the fourth and fifth convolutional layers to obtain the first transformation result.

[0041] The enhanced contour feature map in the original space is transformed through the sixth and seventh convolutional layers to obtain the second transformation result;

[0042] The first transformation result is input into the third adder for calculation to obtain the distance between the instance segmentation label and the contour label, and the first loss function is calculated based on the distance between the instance segmentation label and the contour label.

[0043] The second conversion module is input into the fourth adder for calculation to obtain the distance between the contour label and the instance segmentation label, and the second loss function is calculated based on the distance between the contour label and the instance segmentation label.

[0044] The first loss function and the second loss function are coupled to train the fuzzy image semantic segmentation network, resulting in a trained fuzzy image semantic segmentation network.

[0045] Furthermore, the formula for calculating the distance between the instance segmentation label S and the contour label E is as follows:

[0046]

[0047] Among them, P s P e D represents the feature maps predicted by the semantic segmentation branch and the contour branch, respectively. KL (·) represents the KL divergence calculation formula, C indicates that the task is a C-class classification problem, and c represents the index of each class. This represents the score by which each point on the semantic feature map is predicted to be of class c. This represents the score by which each point on the contour feature map is predicted to be of class c.

[0048] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a semantic segmentation method for traffic scene images in foggy weather.

[0049] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a semantic segmentation method for foggy traffic scene images.

[0050] The above-described solution of the present invention has the following beneficial effects:

[0051] This invention acquires a set of images of traffic scenes in foggy weather; constructs a fuzzy image semantic segmentation network including a dual-task feature extraction module, a mapping module for projecting images from Euclidean space to graph space, a confidence pairing enhancement module for capturing dependencies between feature maps, an anti-mapping module for projecting images from graph space to Euclidean space, and a mutual supervision module; trains the fuzzy image semantic segmentation network by inputting the set of foggy traffic scene images into the fuzzy image semantic segmentation network; and performs semantic segmentation on the target traffic scene image to be processed by inputting it into the trained fuzzy image semantic segmentation network to obtain the segmentation result; compared with the prior art, this invention achieves... The dual-task feature extraction module extracts semantic feature maps and contour feature maps. To flexibly decouple regions with different confidence levels, this invention projects the semantic feature maps and contour feature maps in Euclidean space into semantic feature maps and contour feature maps in graph space through a mapping module, and transmits them to the confidence pairing enhancement module to fully extract the dependencies between feature maps. Through a mutual supervision module, the deep information of the contours of foggy traffic scene images is mined as much as possible to enhance the segmentation accuracy of the blurred image semantic segmentation network. This enables the semantic segmentation accuracy to be improved based on the guidance of peer tasks even when the label data is defective, thereby improving the safety of intelligent driving.

[0052] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the structure of the fuzzy image semantic segmentation network in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the inter-graph reasoning module in an embodiment of the present invention. Detailed Implementation

[0056] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0058] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0059] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0060] This invention addresses existing problems by providing a semantic segmentation method and related equipment for traffic scene images in foggy weather.

[0061] like Figure 1 As shown, an embodiment of the present invention provides a semantic segmentation method for foggy traffic scene images, including:

[0062] Step 1: Obtain a set of images of traffic scenes in foggy weather;

[0063] Step 2, construct a fuzzy image semantic segmentation network, such as Figure 2 As shown, the fuzzy image semantic segmentation model includes a dual-task feature extraction module, a mapping module for projecting the image from Euclidean space to graph space, a confidence pairing enhancement module for capturing the dependencies between feature maps, an anti-mapping module for projecting the image from graph space to Euclidean space, and a mutual supervision module connected in sequence.

[0064] Step 3: Input the foggy traffic scene image set into the fuzzy image semantic segmentation network for training to obtain the trained fuzzy image semantic segmentation network;

[0065] Step 4: Input the target traffic scene image to be processed into the trained fuzzy image semantic segmentation network for semantic segmentation to obtain the segmentation result.

[0066] Specifically, step 3 includes:

[0067] Preprocess the foggy traffic scene image set to obtain the preprocessed foggy traffic scene image set, instance segmentation labels, and contour labels;

[0068] The preprocessed foggy traffic scene image set, instance segmentation labels, and contour labels are input into the fuzzy image semantic segmentation network for training, resulting in the trained fuzzy image semantic segmentation network.

[0069] Specifically, the dual-task feature extraction module includes:

[0070] First residual block, second residual block, third residual block, fourth residual block, first dilated convolutional layer with dilation rate D of 12, second dilated convolutional layer with dilation rate D of 24, third dilated convolutional layer with dilation rate D of 36, first convolutional layer and splicing layer.

[0071] The input of the first residual block is the input of the fuzzy image semantic segmentation network;

[0072] The output of the first residual block is connected to the input of the second residual block. The output of the second residual block is connected to the input of the third residual block. The output of the third residual block is connected to the input of the fourth residual block. The output of the fourth residual block is connected to the first input of the mapping module, the input of the first dilated convolutional layer, the input of the second dilated convolutional layer, the input of the third dilated convolutional layer, and the input of the first convolutional layer, respectively. The outputs of the first dilated convolutional layer, the second dilated convolutional layer, the third dilated convolutional layer, and the first convolutional layer are all connected to the input of the splicing layer. The output of the splicing layer is connected to the second input of the mapping module.

[0073] Preprocessed foggy traffic scene image set Input dual-task feature module, with Θ res The parameters are used to extract basic features through the first residual block, the second residual block, the third residual block, and the fourth residual block, resulting in the semantic segmentation task. First semantic feature map and edge detection task The first contour feature map;

[0074] With Θ aspp Using the parameters, the first semantic feature map is input into the first dilated convolutional layer, the second dilated convolutional layer, the third dilated convolutional layer, and the first convolutional layer respectively to perform multi-scale feature extraction, thereby obtaining semantic feature maps of multiple scales;

[0075] The semantic feature maps of multiple scales, the first semantic feature map, and the first contour feature map are input into the concatenation layer and fused to obtain the second semantic feature map that fuses semantic features and contour features.

[0076] The first contour feature map and the second semantic feature map are input into the mapping module for projection to obtain the contour map and semantic map in the graph space.

[0077] First contour feature map R e Represented as:

[0078]

[0079] Second semantic feature map R s Represented as:

[0080]

[0081] in, Θ represents the feature extraction function. res Θ represents the learnable parameter of the basic feature extraction part in the feature extraction function. aspp This represents the learnable parameters of the multi-scale feature extraction part in the feature extraction function.

[0082] The interweaving of top-down pathways and lateral connections in the dual-task feature extraction module enables it to create high-resolution feature maps from rich semantic layers, resulting in stronger representations. Furthermore, the use of dilated convolutions with different dilation ratios in the interwoven dilated convolutional layers through lateral connections significantly improves the ability to detect objects of different scales in foggy conditions. This allows the dual-task feature extraction module to deeply integrate multi-level features while preserving more semantic details without requiring additional computation.

[0083] In this embodiment of the invention, the degree of degradation of each part of the image is different in foggy weather. In order to flexibly perform subsequent calculation processing on any block in the image, this embodiment of the invention projects the feature map from Euclidean space to graph space. Based on this transformation, regions with different degrees of influence can be decoupled.

[0084] Specifically, the mapping module is implemented through the following mathematical calculations. The key to graph mapping lies in determining the vertex features of the mapped graph. Each vertex feature is derived from the features of certain points on the feature map, r. i It is a vertex on the feature map, v k These are the characteristics of the vertices in the transformed graph. The transformation method is shown in the following three formulas.

[0085] In terms of form, the embodiments of the present invention use 1×1 convolutional kernels to reduce the dimensionality of the first contour feature map and the second semantic feature map, thereby achieving low-dimensional representation. Then, a mapping module is used to transform the first contour feature map and the second semantic feature map into contour maps that depend on the samples. semantic graph After conversion, the outline drawing semantic graph In the graph, vertices represent pixels with similar features, while edges are used to measure the proximity between different vertices in the graph feature space; embodiments of the present invention... and The mapping module is parameterized, where W and ∑ are learnable parameters, and is the number of vertices. After conversion into a graph, the features of each node can be calculated using the following formula:

[0086]

[0087] Where, r i w k and σ k R respectively l The column vectors of , W and ∑, where " / " indicates element-wise division, each v′ k Represents the eigenvector r i With vertex parameter w k The weighted average of the residuals Represents the eigenvector r i For w k The soft allocation is defined by the following formula:

[0088]

[0089] In addition, for v′ k L2 normalization is performed to calculate the final vertex feature vector v. k :

[0090]

[0091] After collecting all the v k Subsequently, the feature matrix can be finally obtained in the embodiments of the present invention. This feature matrix contains K columns, where c represents the feature length. Correspondingly, the semantic segmentation task... and edge detection tasks The vertex features are V s and V e .

[0092] Specifically, the second step in graph mapping is to determine the edges of the mapped graph. This embodiment of the invention considers constructing a weighted complete graph (i.e., there is a weighted edge between any two vertices). Therefore, the weights are obtained by the following formula. The resulting graph has K vertices. The i-th row and j-th column of the K×K parameters in the adjacency matrix represents the weight information between the i-th and j-th vertices, and the adjacency matrix is ​​a symmetric matrix.

[0093] The embodiments of the present invention require further semantic segmentation tasks. and edge detection tasks The adjacency matrix is ​​calculated using vertex features, which prepares the subsequent confidence pairing enhancement module.

[0094] In-graph adjacency matrix Normalization operation To obtain, among which The SoftMax operation was used, and the corresponding method for calculating the graph adjacency matrix is ​​as follows:

[0095]

[0096]

[0097] After projection and It was used to represent semantics and contour maps, where, and It consists of K vertices. and V e and V s The column vectors are such that both the mapping and anti-mapping phases can be learned.

[0098] Specifically, the confidence-matching enhancement module includes:

[0099] First inter-graph reasoning module, second inter-graph reasoning module, intra-graph reasoning module;

[0100] The first input terminal of the first inter-graph reasoning module, the first input terminal of the second inter-graph reasoning module, and the second input terminal of the second inter-graph reasoning module are all connected to the first output terminal of the mapping module. The second input terminal and the third input terminal of the first inter-graph reasoning module are all connected to the second output terminal of the mapping module. The output terminal of the first inter-graph reasoning module is connected to the third input terminal of the second inter-graph reasoning module and the second input terminal of the intra-graph reasoning module, respectively. The output terminal of the second inter-graph reasoning module is connected to the first input terminal of the intra-graph reasoning module. The first output terminal of the intra-graph reasoning module is connected to the first input terminal of the anti-mapping module. The second output terminal of the intra-graph reasoning module is connected to the second input terminal of the anti-mapping module.

[0101] The contour map and semantic map are input into the confidence pairing enhancement module. The first inter-graph reasoning module performs cross-reasoning on the semantic map and the contour map to capture the dependency relationship between the semantic map and the contour map, and obtains the enhanced semantic map.

[0102] The second inter-graph reasoning module performs cross-reasoning on the enhanced semantic graph and the contour graph to capture the dependency relationship between the enhanced semantic graph and the contour graph, and obtains the enhanced contour graph.

[0103] The enhanced semantic map and the enhanced contour map are input into the in-graph reasoning module for in-graph feature extraction, resulting in the enhanced semantic feature map and the enhanced contour feature map.

[0104] The enhanced semantic feature map and the enhanced contour feature map are input into the inverse mapping module for inverse projection to obtain the enhanced semantic feature map and the enhanced contour feature map in the original space.

[0105] Specifically, such as Figure 3 As shown, both the first inter-graph reasoning module and the second inter-graph reasoning module include:

[0106] The system consists of a second convolutional layer, a third convolutional layer, a first multilayer perceptron, a second multilayer perceptron, a third multilayer perceptron, a first multiplier, a first adder, a second multiplier, a second adder, and a position compensator for compensating the position vector.

[0107] The input of the second convolutional layer is connected to the first output of the mapping module and the first input of the second adder, respectively. The output of the second convolutional layer is connected to the input of the first multilayer perceptron. The output of the first multilayer perceptron is connected to the first input of the first multiplier and the first input of the second adder, respectively. The second output of the first multilayer perceptron is connected to the first input of the first multiplier.

[0108] The input of the third convolutional layer is connected to the second output of the mapping module. The output of the third convolutional layer is connected to the inputs of the second and third multilayer perceptrons. The output of the second multilayer perceptron is connected to the second input of the first multiplier. The outputs of the second multiplier and the position compensator are both connected to the input of the first adder. The output of the first adder is connected to the first input of the second multiplier. The output of the third multilayer perceptron is connected to the second input of the second multiplier. The output of the second multiplier is connected to the second input of the second adder. The output of the second adder is connected to the input of the in-graph inference module.

[0109] In this embodiment of the invention, the inter-graph reasoning module aims to use the drift calculation function f DC and feature aggregation function To infer the dependencies between the semantic graph and the contour graph respectively, the drift calculation function includes a contour-oriented drift calculation function. and semantically oriented drift computation function

[0110] This invention takes contour-oriented drift calculation as an example, using a semantic map and a contour map as input, and further stacking the vertices to form a two-dimensional vector. The semantic map and contour map are then fed into three different multilayer perceptrons. Transform into features and characteristics Will Transform into features The drift matrix facing the contour is then calculated using the following formula.

[0111]

[0112]

[0113] in, and It is a learnable parameter matrix, B c This represents a global positional bias used to compensate for the loss of positional information during image projection, and then the function... Used to calculate normalized weights; finally, according to the formula for the inter-graph adjacency matrix, the drift matrix facing the contour can be obtained as follows:

[0114]

[0115] Then, through feature aggregation function The semantic information is transformed from a semantic graph to a contour graph, thus obtaining an enhanced contour graph. This process is represented by the following formula:

[0116]

[0117] Wherein, α is used to control the relationship with respect to V e The semantic information propagation weighting parameters.

[0118] Specifically, the in-graph reasoning module includes:

[0119] First graph convolutional network and second graph convolutional network;

[0120] The input of the first graph convolutional network is connected to the output of the second adder in the first graph inference module, and the output of the first graph convolutional network is connected to the second input of the inverse mapping module.

[0121] The input of the second graph convolutional network is connected to the output of the second adder in the second graph inference module, and the output of the second graph convolutional network is connected to the first input of the inverse mapping module.

[0122] The main goal of the in-graph inference module is to exchange information between each node in the graph, helping to improve the prediction accuracy of some uncertain vertices. and Let W represent the K×K adjacency matrices of the semantic graph and the contour graph, respectively. (t) Indicates the t-th th The learnable parameters in the layers are used in the graph inference module of this embodiment to extract graph features using a graph convolutional network, as shown in the following formula:

[0123]

[0124] Among them, W s and W e It is a randomly initialized weight matrix that is learned through gradient descent during training. σ(·) represents a non-linear activation function, and I represents the identity matrix, which is used to reduce the difficulty of convergence.

[0125] In order to make the model provided by this invention more compatible with existing convolutional neural network-based frameworks, the enhanced feature maps are reprojected back into Euclidean space through an inverse mapping module. Therefore, the graph assignment matrices of the contour map and the semantic map are represented as follows: and Where k∈{0,…,K-1}, i∈{0,…,wh-1}, the inverse mapping process of the graph can be described as:

[0126]

[0127] in and and These represent the original features of the contour branch and the semantic segmentation branch, respectively. Through the above operations, the enhanced semantic feature map and contour feature map in the original space can be obtained.

[0128] Specifically, the mutual supervision module includes:

[0129] Fourth convolutional layer, fifth convolutional layer, sixth convolutional layer, seventh convolutional layer, third adder and fourth adder;

[0130] The input terminals of the fourth and fifth convolutional layers are both connected to the first output terminal of the inverse mapping module. The output terminal of the fourth convolutional layer is connected to the first input terminal of the third adder and the output terminal of the sixth convolutional layer, respectively. The second input terminal of the third adder is connected to the output terminal of the sixth convolutional layer. The input terminals of the sixth and seventh convolutional layers are both connected to the second output terminal of the inverse mapping module. The output terminal of the seventh convolutional layer is connected to the first input terminal of the fourth adder and the output terminal of the fifth convolutional layer, respectively. The second input terminal of the fourth adder is connected to the output terminal of the fifth convolutional layer.

[0131] The enhanced semantic feature map in the original space, the enhanced contour feature map in the original space, the contour label and the instance segmentation label are all input into the mutual supervision module. The enhanced semantic feature map in the original space is transformed through the fourth and fifth convolutional layers to obtain the first transformation result.

[0132] The enhanced contour feature map in the original space is transformed through the sixth and seventh convolutional layers to obtain the second transformation result;

[0133] The first transformation result is input into the third adder for calculation to obtain the distance between the instance segmentation label and the contour label, and the first loss function is calculated based on the distance between the instance segmentation label and the contour label.

[0134] The second conversion module is input into the fourth adder for calculation to obtain the distance between the contour label and the instance segmentation label, and the second loss function is calculated based on the distance between the contour label and the instance segmentation label.

[0135] The first loss function and the second loss function are coupled to train the fuzzy image semantic segmentation network, resulting in a trained fuzzy image semantic segmentation network.

[0136] In this embodiment of the invention, the enhanced semantic features and contour features are further mapped to... and Among them, C s This indicates the number of categories, while C e The value is 2.

[0137] During training, this embodiment of the invention employs bilinear interpolation to resize the output to the original input size in order to calculate the loss:

[0138]

[0139] in, It is the true label, when the prediction is correct. Indicator Function For p c Taking semantic segmentation branches as an example,

[0140] Unlike traditional training methods, this embodiment of the invention uses a mutual supervision module to simultaneously enhance two tasks. To quantify the dissimilarity between the two branches, KL divergence is used to calculate the distribution difference. However, due to the inherent differences between the two task domains, directly minimizing the KL divergence of the predictions for the two branches is not suitable. Therefore, two additional convolutional layers are used to combine the predicted values ​​P. e Convert to P sConvert to This represents the domain transformation. Then, the distance from task S to E is calculated as follows:

[0141]

[0142] Among them, P s P e D represents the feature maps predicted by the semantic segmentation branch and the contour branch, respectively. KL (·) represents the KL divergence calculation formula, C indicates that the task is a C-class classification problem, and c represents the index of each class. This represents the score by which each point on the semantic feature map is predicted to be of class c. This represents the score by which each point on the contour feature map is predicted to be of class c.

[0143] Therefore, the overall loss function is as follows:

[0144]

[0145]

[0146] Based on the two loss functions mentioned above, the skill learns the true label and can also learn the estimated value from its peer branches based on the KL divergence, thereby effectively increasing the accuracy of instance segmentation in foggy conditions.

[0147] This invention acquires a set of foggy traffic scene images; constructs a fuzzy image semantic segmentation network including a dual-task feature extraction module, a mapping module for projecting images from Euclidean space to graph space, a confidence pairing enhancement module for capturing dependencies between feature maps, an anti-mapping module for projecting images from graph space to Euclidean space, and a mutual supervision module; trains the foggy traffic scene image set into the fuzzy image semantic segmentation network to obtain the trained fuzzy image semantic segmentation network; and inputs the target traffic scene image to be processed into the trained fuzzy image semantic segmentation network for semantic segmentation to obtain the segmentation result. Compared with the prior art, this invention... The invention extracts semantic feature maps and contour feature maps through a dual-task feature extraction module. To flexibly decouple regions with different confidence levels, the invention projects the semantic feature maps and contour feature maps in Euclidean space into semantic feature maps and contour feature maps in graph space through a mapping module. These maps are then transmitted to the confidence pairing enhancement module to fully extract the dependencies between the feature maps. The mutual supervision module further mines the deep-level information of the contours in foggy traffic scene images to enhance the segmentation accuracy of the blurred image semantic segmentation network. This enables the semantic segmentation accuracy to be improved even when the label data is defective, based on the guidance of peer tasks, thereby improving the safety of intelligent driving.

[0148] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a semantic segmentation method for foggy traffic scene images.

[0149] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention can implement all or part of the processes in the methods described above by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a building device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0150] This invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a semantic segmentation method for foggy traffic scene images.

[0151] It should be noted that the terminal device can be a mobile phone, tablet computer, laptop computer, Ultra-mobile Personal Computer (UMPC), netbook, Personal Digital Assistant (PDA), etc. For example, the terminal device can be a station (ST) in a WLAN, a cellular phone, cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, PDA, handheld device with wireless communication capabilities, computing device or other processing device connected to a wireless modem, computer, laptop computer, handheld communication device, handheld computing device, satellite wireless device, etc. The embodiments of this invention do not limit the specific type of terminal device.

[0152] The processor referred to can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0153] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital Card (SD), or Flash Card. Furthermore, the memory may include both internal and external storage units of the terminal device. The memory is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0154] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0155] The above describes a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for traffic scene images in foggy weather, characterized in that, include: Step 1: Obtain a set of images of traffic scenes in foggy weather; Step 2: Construct a fuzzy image semantic segmentation network. The fuzzy image semantic segmentation model includes a dual-task feature extraction module, a mapping module for projecting the image from Euclidean space to graph space, a confidence pairing enhancement module for capturing the dependencies between feature maps, an anti-mapping module for projecting the image from graph space to Euclidean space, and a mutual supervision module, which are connected in sequence. Step 3: Input the set of foggy traffic scene images into the fuzzy image semantic segmentation network for training to obtain the trained fuzzy image semantic segmentation network; Step 4: Input the target traffic scene image to be processed into the trained fuzzy image semantic segmentation network for semantic segmentation to obtain the segmentation result; The dual-task feature extraction module includes: a first residual block, a second residual block, a third residual block, a fourth residual block, a first dilated convolutional layer, a second dilated convolutional layer, a third dilated convolutional layer, a first convolutional layer, and a stitching layer. The preprocessed foggy traffic scene image set is input into the dual-task feature module. Basic features are extracted through the first residual block, the second residual block, the third residual block, and the fourth residual block to obtain a first semantic feature map and a first contour feature map. The first semantic feature map is then input into the first dilated convolutional layer, the second dilated convolutional layer, the third dilated convolutional layer, and the first convolutional layer to obtain semantic feature maps at multiple scales. The semantic feature maps at multiple scales, the first semantic feature map, and the first contour feature map are input into the stitching layer for fusion to obtain a second semantic feature map that fuses semantic and contour features. The first contour feature map and the second semantic feature map are then input into the mapping module for projection to obtain a contour map and a semantic map in graph space. The confidence pairing enhancement module includes: a first inter-graph reasoning module, a second inter-graph reasoning module, and an intra-graph reasoning module. The contour map and the semantic map are input into the confidence pairing enhancement module. The first inter-graph reasoning module performs cross-reasoning on the semantic map and the contour map to capture the dependencies between them, resulting in an enhanced semantic map. The second inter-graph reasoning module performs cross-reasoning on the enhanced semantic map and the contour map to capture the dependencies between them, resulting in an enhanced contour map. The enhanced semantic map and the enhanced contour map are input into the intra-graph reasoning module for intra-graph feature extraction, resulting in an enhanced semantic feature map and an enhanced contour feature map. The enhanced semantic feature map and the enhanced contour feature map are input into the inverse mapping module for inverse projection, resulting in an enhanced semantic feature map and an enhanced contour feature map in the original space.

2. The semantic segmentation method for foggy traffic scene images according to claim 1, characterized in that, Step 3 includes: The foggy traffic scene image set is preprocessed to obtain the preprocessed foggy traffic scene image set, instance segmentation labels, and contour labels. The preprocessed foggy traffic scene image set, the instance segmentation labels, and the contour labels are input into the fuzzy image semantic segmentation network for training, resulting in the trained fuzzy image semantic segmentation network.

3. The semantic segmentation method for foggy traffic scene images according to claim 2, characterized in that, The dual-task feature extraction module includes: The input terminal of the first residual block is the input terminal of the fuzzy image semantic segmentation network; The output of the first residual block is connected to the input of the second residual block, the output of the second residual block is connected to the input of the third residual block, the output of the third residual block is connected to the input of the fourth residual block, the output of the fourth residual block is connected to the first input of the mapping module, the input of the first dilated convolutional layer, the input of the second dilated convolutional layer, the input of the third dilated convolutional layer, and the input of the first convolutional layer, respectively. The outputs of the first dilated convolutional layer, the second dilated convolutional layer, the third dilated convolutional layer, and the first convolutional layer are all connected to the input of the splicing layer, and the output of the splicing layer is connected to the second input of the mapping module.

4. The semantic segmentation method for foggy traffic scene images according to claim 3, characterized in that, The confidence pairing enhancement module includes: The first input terminal of the first inter-graph inference module, the first input terminal of the second inter-graph inference module, and the second input terminal of the second inter-graph inference module are all connected to the first output terminal of the mapping module. The second input terminal of the first inter-graph inference module and the third input terminal of the first inter-graph inference module are all connected to the second output terminal of the mapping module. The output terminal of the first inter-graph inference module is connected to the third input terminal of the second inter-graph inference module and the second input terminal of the intra-graph inference module, respectively. The output terminal of the second inter-graph inference module is connected to the first input terminal of the intra-graph inference module. The first output terminal of the intra-graph inference module is connected to the first input terminal of the anti-mapping module. The second output terminal of the intra-graph inference module is connected to the second input terminal of the anti-mapping module.

5. The semantic segmentation method for foggy traffic scene images according to claim 4, characterized in that, Both the first inter-graph inference module and the second inter-graph inference module include: The system consists of a second convolutional layer, a third convolutional layer, a first multilayer perceptron, a second multilayer perceptron, a third multilayer perceptron, a first multiplier, a first adder, a second multiplier, a second adder, and a position compensator. The input of the second convolutional layer is connected to the first output of the mapping module and the first input of the second adder, respectively. The output of the second convolutional layer is connected to the input of the first multilayer perceptron. The output of the first multilayer perceptron is connected to the first input of the first multiplier and the first input of the second adder, respectively. The second output of the first multilayer perceptron is connected to the first input of the first multiplier. The input of the third convolutional layer is connected to the second output of the mapping module. The output of the third convolutional layer is connected to the input of the second multilayer perceptron and the input of the third multilayer perceptron. The output of the second multilayer perceptron is connected to the second input of the first multiplier. The output of the second multiplier and the output of the position compensator are both connected to the input of the first adder. The output of the first adder is connected to the first input of the second multiplier. The output of the third multilayer perceptron is connected to the second input of the second multiplier. The output of the second multiplier is connected to the second input of the second adder. The output of the second adder is connected to the input of the in-graph inference module.

6. The semantic segmentation method for foggy traffic scene images according to claim 5, characterized in that, The in-graph reasoning module includes: First graph convolutional network and second graph convolutional network; The input of the first graph convolutional network is connected to the output of the second adder in the first graph inference module, and the output of the first graph convolutional network is connected to the second input of the inverse mapping module. The input of the second graph convolutional network is connected to the output of the second adder in the second graph inference module, and the output of the second graph convolutional network is connected to the first input of the inverse mapping module.

7. The semantic segmentation method for foggy traffic scene images according to claim 6, characterized in that, The mutual supervision module includes: Fourth convolutional layer, fifth convolutional layer, sixth convolutional layer, seventh convolutional layer, third adder and fourth adder; The input terminals of the fourth and fifth convolutional layers are both connected to the first output terminal of the inverse mapping module. The output terminal of the fourth convolutional layer is connected to the first input terminal of the third adder and the output terminal of the sixth convolutional layer, respectively. The second input terminal of the third adder is connected to the output terminal of the sixth convolutional layer. The input terminals of the sixth and seventh convolutional layers are both connected to the second output terminal of the inverse mapping module. The output terminal of the seventh convolutional layer is connected to the first input terminal of the fourth adder and the output terminal of the fifth convolutional layer, respectively. The second input terminal of the fourth adder is connected to the output terminal of the fifth convolutional layer. The enhanced semantic feature map in the original space, the enhanced contour feature map in the original space, the contour label and the instance segmentation label are all input into the mutual supervision module. The enhanced semantic feature map in the original space is transformed through the fourth convolutional layer and the fifth convolutional layer to obtain the first transformation result. The enhanced contour feature map in the original space is transformed by the sixth and seventh convolutional layers to obtain the second transformation result; The first conversion result is input into the third adder for calculation to obtain the distance between the instance segmentation label and the contour label, and the first loss function is calculated based on the distance between the instance segmentation label and the contour label; The second conversion module is input into the fourth adder for calculation to obtain the distance between the contour label and the instance segmentation label, and the second loss function is calculated based on the distance between the contour label and the instance segmentation label; The first loss function and the second loss function are coupled to train the fuzzy image semantic segmentation network, resulting in a trained fuzzy image semantic segmentation network.

8. The semantic segmentation method for foggy traffic scene images according to claim 7, characterized in that, The formula for calculating the distance between the instance segmentation label S and the contour label E is as follows: ； in, , These represent the feature maps predicted by the semantic segmentation branch and the contour branch, respectively. This represents the formula for calculating KL divergence. This indicates that the task is a C classification problem. This represents the index of each category. This indicates that each point on the semantic feature map is predicted as the first... Class fractions, This indicates that each point on the contour feature map is predicted as the first... The score of the class.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the semantic segmentation method for foggy traffic scene images as described in any one of claims 1 to 8.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the semantic segmentation method for foggy traffic scene images as described in any one of claims 1 to 8.

Images