Lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search

Abstract

The present application relates to the field of deep learning technology, and in particular to a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search, comprising: constructing and pre-training a dual-branch lightweight backbone network; constructing a hierarchical unit-level search space suitable for RGB-D image semantic segmentation, and finally obtaining a cross-modal multi-scale feature adaptive fusion network; inputting the outputs of the third, fourth, and fifth layers of the dual-branch lightweight backbone network into the cross-modal multi-scale feature adaptive fusion network for search, and optimizing using a gradient descent optimization algorithm to obtain the optimal cross-modal multi-scale feature adaptive fusion network; inputting the outputs of the third, fourth, and fifth layers of the dual-branch lightweight backbone network into the optimal cross-modal multi-scale feature adaptive fusion network for retraining, and obtaining the semantic segmentation result of the multi-modal input image. The method achieves a balance between semantic segmentation accuracy and speed.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of deep learning technology, and in particular to a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search. Background Art

[0002] Due to the success of deep convolutional neural networks, semantic segmentation has made significant progress. Considering the increasing demand for real-time semantic segmentation in robot navigation, autonomous driving, etc., real-time semantic segmentation has become a hot topic in recent years. Compared with RGB images, depth images can provide rich three-dimensional geometric information, which makes the semantic segmentation of RGB-D images show good robustness in the face of scenes with high foreground and background confusion and insufficient light. Therefore, more and more researchers and teams are committed to conducting research on semantic segmentation of RGB-D images. However, most of the methods proposed so far consume a lot of computational costs, which poses a challenge to mobile platforms that require instant response in reality. Therefore, how to automatically search and design efficient neural network architectures suitable for different tasks has become a hot issue at present.

[0003] The network architectures used in the currently proposed RGB-D image semantic segmentation methods include manually designed networks and networks based on Neural Architecture Search (NAS). Sun et al. used ResNet18 as an encoder and combined it with an efficient upsampling module to improve the real-time reasoning capability of semantic segmentation. Liu et al. introduced NAS into semantic segmentation. The role of NAS is to search for unit structures and connection operations between two nodes. Wu et al. designed a multi-resolution branch obtained by NAS search for real-time semantic segmentation.

[0004] However, the above methods still have some limitations and shortcomings. First, most of the above methods follow a fixed paradigm, that is, searching for an optimal unit structure, and the final network architecture is composed of multiple identical optimal units stacked together. Although searching for a fixed unit structure can increase the search speed, this ignores the changing requirements for different tasks in the semantic segmentation task. Second, the above methods only use one unit structure for stacking, which cannot fully utilize contextual information of different scales. The strategy of using the same unit for stacking to form the final network is not the optimal construction strategy. Third, the above multimodal semantic segmentation model cannot achieve a good balance between the pursuit of accuracy and speed. Summary of the invention

[0005] In view of this, an embodiment of the present application proposes a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search, which fully considers the feature extraction problem of feature maps of different resolutions, enhances the robustness of RGB-D real-time image semantic segmentation, and thus achieves a good balance between semantic segmentation accuracy and semantic segmentation speed.

[0006] In order to achieve the above-mentioned purpose, an embodiment of the present application proposes a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search, the method comprising the following steps: constructing and pre-training a dual-branch lightweight backbone network, the dual-branch lightweight backbone network comprising an RGB backbone network and a deep backbone network, each backbone network being divided into five layers, including a first layer and a second layer for fast downsampling and shallow feature extraction, and a third layer, a fourth layer and a fifth layer provided with a residual structure for high-level semantic feature extraction; constructing a hierarchical unit-level search space suitable for RGB-D image semantic segmentation, and obtaining a cross-modal multi-scale feature adaptive fusion network, the cross-modal multi-scale feature adaptive fusion network consisting of two RGB top units, two RGB middle units, two RGB bottom units, two deep top units, two deep middle units, two deep bottom units and three dynamic fusion modules; wherein feature maps of the same resolution share the same unit-level search space, and feature maps of different resolutions have different unit-level search spaces; the RGB image and the depth image in the multimodal input image are subjected to a 3×3 convolution operation, and the fusion network is ... After multiplication processing, they are input into the RGB backbone network and the deep backbone network respectively. The outputs of the third, fourth and fifth layers of the RGB backbone network and the third, fourth and fifth layers of the deep backbone network are input into the cross-modal multi-scale feature adaptive fusion network for search and optimization using the gradient descent optimization algorithm. After decoding, the optimal cross-modal multi-scale feature adaptive fusion network is obtained; the outputs of the third, fourth and fifth layers of the RGB backbone network and the third, fourth and fifth layers of the deep backbone network are input into the optimal cross-modal multi-scale feature adaptive fusion network for retraining to obtain the final cross-modal multi-scale feature adaptive fusion network; the output of the third dynamic fusion module of the final cross-modal multi-scale feature adaptive fusion network is up-sampled and spliced ​​with the output of the second dynamic fusion module in the channel dimension, and then the features obtained after splicing are up-sampled and spliced ​​with the output of the first dynamic fusion module in the channel dimension, and then up-sampled and a 3×3 convolution are processed to obtain the semantic segmentation result of the multimodal input image.

[0007] In order to achieve the above-mentioned purpose, the embodiment of the present application also proposes a lightweight RGB-D image semantic segmentation system based on hierarchical shared architecture search, wherein the system is composed of a dual-branch lightweight backbone network construction module, a cross-modal multi-scale feature adaptive fusion network construction module, an optimization module, a final training module and an execution module; the dual-branch lightweight backbone network construction module is used to construct and pre-train a dual-branch lightweight backbone network, the dual-branch lightweight backbone network includes an RGB backbone network and a deep backbone network, each backbone network is divided into five layers, including a first layer and a second layer for fast downsampling and shallow feature extraction, and a residual structure is set The third, fourth and fifth layers are used for high-level semantic feature extraction; a cross-modal multi-scale feature adaptive fusion network construction module is used to construct a hierarchical unit-level search space suitable for RGB-D image semantic segmentation, and a cross-modal multi-scale feature adaptive fusion network is obtained. The cross-modal multi-scale feature adaptive fusion network consists of two RGB top units, two RGB middle units, two RGB bottom units, two deep top units, two deep middle units, two deep bottom units and three dynamic fusion modules, among which feature maps of the same resolution share the same unit-level search space, and feature maps of different resolutions have different unit-level search spaces. The optimization module is used to process the RGB image and the depth image in the multimodal input image through 3×3 convolution, and then input them into the RGB backbone network and the deep backbone network respectively. The outputs of the third, fourth and fifth layers of the RGB backbone network and the third, fourth and fifth layers of the deep backbone network are input into the cross-modal multi-scale feature adaptive fusion network for search, and the gradient descent optimization algorithm is used for optimization. After decoding, the optimal cross-modal multi-scale feature adaptive fusion network is obtained. The final training module is used to input the outputs of the third, fourth and fifth layers of the RGB backbone network and the third, fourth and fifth layers of the deep backbone network into the cross-modal multi-scale feature adaptive fusion network for search, and the gradient descent optimization algorithm is used for optimization. After decoding, the optimal cross-modal multi-scale feature adaptive fusion network is obtained. The outputs of the third, fourth and fifth layers of the backbone network are input into the optimal cross-modal multi-scale feature adaptive fusion network for retraining to obtain the final cross-modal multi-scale feature adaptive fusion network; an execution module is used to upsample the output of the third dynamic fusion module of the final cross-modal multi-scale feature adaptive fusion network, splice it with the output of the second dynamic fusion module in the channel dimension, and then upsample the features obtained after splicing, splice it with the output of the first dynamic fusion module in the channel dimension, and then obtain the semantic segmentation result of the multimodal input image after upsampling and a 3×3 convolution.

[0008] In order to achieve the above-mentioned purpose, an embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search.

[0009] In order to achieve the above-mentioned purpose, an embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the above-mentioned lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search.

[0010] The embodiment of the present application proposes a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search, constructs and pre-trains a dual-branch lightweight backbone network, the RGB backbone network in the dual-branch lightweight backbone network is used to extract features from RGB images, and the deep backbone network is used to extract features from deep images, which can fully explore shallow features and high-level semantic features in multimodal images. A hierarchical unit-level search space suitable for RGB-D image semantic segmentation is constructed to obtain a cross-modal multi-scale feature adaptive fusion network. Feature maps of the same resolution share the same unit-level search space, and feature maps of different resolutions have different unit-level search spaces, thereby performing appropriate feature extraction on feature maps of different resolutions. After search optimization and final training, the cross-modal multi-scale feature adaptive fusion network can be used for semantic segmentation tasks, which enhances the robustness of RGB-D real-time image semantic segmentation, thereby achieving a good balance between semantic segmentation accuracy and semantic segmentation speed, and providing reliable data support for applications based on semantic segmentation.

[0011] Optionally, the first and second layers of the backbone network are both ordinary convolutional layers, each layer contains two 3×3 convolutions, the width and height of the outputs of the first and second layers of the backbone network are both half of the input, and the number of channels of the outputs of the first and second layers of the backbone network are both twice that of the input; the third, fourth and fifth layers of the backbone network are each a lightweight module, and the lightweight module is formed by stacking a first sub-network and two second sub-networks, the first sub-network includes a regular branch and a residual branch, and the regular branch of the first sub-network is composed of 3×3 convolutions, It consists of 3×3 depth-separable convolutions with a stride of 2, 3×3 depth-separable convolutions, and 1×1 convolutions. The residual branch of the first subnetwork consists of 3×3 depth-separable convolutions with a stride of 2 and 1×1 convolutions. The output of the first subnetwork is the pixel-by-pixel addition of the output of the regular branch of the first subnetwork and the output of the residual branch of the first subnetwork. The second subnetwork consists of 3×3 convolutions, 3×3 depth-separable convolutions, and 1×1 convolutions. The output of the second subnetwork is the pixel-by-pixel addition of the output of the 1×1 convolution of the second subnetwork and the input of the second subnetwork.

[0012] Optionally, in the cross-modal multi-scale feature adaptive fusion network, the input of the second RGB top unit is the output of the first RGB top unit and the output of the first RGB middle unit, the input of the second RGB middle unit is the output of the first RGB middle unit and the output of the first RGB bottom unit, the input of the second RGB bottom unit is the output of the first RGB top unit and the output of the first RGB bottom unit, the input of the second depth top unit is the output of the first depth top unit and the output of the first depth middle unit, the input of the second depth middle unit is the output of the first depth middle unit and the output of the first depth bottom unit, the input of the second depth bottom unit is the output of the first depth top unit and the output of the first depth bottom unit, the input of the third dynamic fusion module is the output of the second RGB bottom unit, the output of the second depth bottom unit, and the result of element-by-element addition of the output of the second RGB bottom unit and the output of the second depth bottom unit, the input of the second dynamic fusion module is the output of the second RGB middle unit, the output of the second depth middle unit, and the output of the third dynamic fusion module, and the input of the first dynamic fusion module is the output of the second RGB top unit, the output of the second depth top unit, and the output of the second dynamic fusion module.

[0013] Optionally, the outputs of the third, fourth and fifth layers of the RGB backbone network, and the outputs of the third, fourth and fifth layers of the deep backbone network, are input into a cross-modal multi-scale feature adaptive fusion network for searching, including: using the outputs of the third and fourth layers of the RGB backbone network as the input of the first RGB top unit, using the outputs of the fourth and fifth layers of the RGB backbone network as the input of the first RGB middle unit, using the outputs of the third and fifth layers of the RGB backbone network as the input of the first RGB bottom unit, using the outputs of the third and fourth layers of the deep backbone network as the input of the first deep top unit, using the outputs of the fourth and fifth layers of the deep backbone network as the input of the first deep middle unit, and using the outputs of the third and fifth layers of the deep backbone network as the input of the first deep bottom unit, thereby searching.

[0014] Optionally, the structures of the RGB top unit, the RGB middle unit and the RGB bottom unit in the cross-modal multi-scale feature adaptive fusion network are different, and the structures of the depth top unit, the depth middle unit and the depth bottom unit are different, each unit is composed of four nodes, including two input nodes, one middle node and one output node, the edge between the nodes is an operation in the operation set, and the search goal is to find the best operation on each edge; the operation set is composed of three types of operations, namely non-learning operations, standard convolutions and dilated convolutions, the non-learning operations include global average pooling, global maximum pooling and jump connections, global average pooling includes 3×3 global average pooling and 5×5 global average pooling, global maximum pooling includes 3×3 global maximum pooling and 5×5 global maximum pooling, standard convolution includes 3×3 standard convolution, 5×5 standard convolution and 7×7 standard convolution, and dilated convolution includes dilated convolution with a dilation rate of 2, dilated convolution with a dilation rate of 4 and dilated convolution with a dilation rate of 8.

[0015] Optionally, for any unit in the cross-modal multi-scale feature adaptive fusion network, let its two input nodes be and The intermediate node is , the output node is , and It is also used to represent the output of two input nodes. It is also used to represent the output of the intermediate node. It is also used to represent the output of an output node and the input node Output and input nodes The output of each operation is respectively After the operations in the process are processed, they are transmitted to the intermediate nodes. , intermediate node The outputs of the two input nodes are summed and activated by the Relu activation function. The output of After processing, it is transferred to the output node , output node The output is expressed by the formula:

[0016] ;

[0017] ;

[0018] ;

[0019] ;

[0020] in, For operation collection The total number of operations in Represents a set of operations The operations, express The set of operations on this edge The The weighting factor of the operation, express The set of operations on this edge The The normalized weight coefficients of the operations, express The set of operations on this edge The The weighting factor of the operation, express The set of operations on this edge The The normalized weight coefficients of the operations, express The set of operations on this edge The The weighting factor of the operation, express The set of operations on this edge The The normalized weight coefficients of the operations.

[0021] Optionally, for the third dynamic fusion module, its three inputs are , and , and and the output of the second RGB bottom unit and the output of the second depth bottom unit, respectively, is the result of element-by-element addition of the output of the second RGB bottom unit and the output of the second depth bottom unit; the third dynamic fusion module consists of an RGB processing subnetwork, a depth processing subnetwork, a fusion subnetwork and a projection subnetwork, the RGB processing subnetwork consists of a first RGB processing branch, a second RGB processing branch and a first reshaping unit, the depth processing subnetwork consists of a first depth processing branch, a second depth processing branch and a second reshaping unit, the first RGB processing branch and the first depth processing branch are both a 1×1 convolution layer, the second RGB processing branch and the second depth processing branch are both composed of two consecutive 3×3 convolution layers, and each 3×3 convolution layer is followed by a LeakRelu activation function layer; Enter the first RGB processing branch, and After channel splicing, the second RGB processing branch is entered. The output of the first RGB processing branch and the output of the second RGB processing branch are added element by element and reshaped into the first feature by the first reshaping unit. Enter the first depth processing branch, and After channel splicing, it enters the second deep processing branch. The output of the first deep processing branch and the output of the second deep processing branch are added element by element, and then reshaped into the second feature by the second reshaping unit; a gating unit is set in the fusion subnetwork, and the gating unit is set with four control operations: pixel-by-pixel addition, pixel-by-pixel subtraction, channel splicing and Hadamard product. The fusion subnetwork fuses the feature subspace corresponding to the first feature and the feature subspace corresponding to the second feature based on the gating unit to obtain the final feature subspace; the projection subnetwork uses orthogonal linear projection to Project it into the final feature subspace for reconstruction to obtain the reconstructed features As the output of the third dynamic fusion module. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 is a flowchart of a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search provided in one embodiment of the present application;

[0024] Figure 2It is a schematic diagram of the structure of a dual-branch lightweight backbone network and a cross-modal multi-scale feature adaptive fusion network provided in one embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of the structure of the third layer, the fourth layer, the fifth layer, the first sub-network and the second sub-network of the backbone network provided in one embodiment of the present application;

[0026] Figure 4 is a schematic diagram of an input node, an intermediate node, and an output node provided in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of the structure of a third dynamic fusion module provided in an embodiment of the present application;

[0028] Figure 6 is a schematic structural diagram of a fusion unit provided in one embodiment of the present application;

[0029] Figure 7 is a structural schematic diagram of a lightweight RGB-D image semantic segmentation system based on hierarchical shared architecture search provided in another embodiment of the present application;

[0030] Figure 8 It is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0031] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0032] An embodiment of the present application proposes a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search, which is applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments are all described by taking the server as an example. The following is a specific description of the implementation details of a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search proposed in this embodiment. The following content is only the implementation details provided for the convenience of understanding and is not necessary for the implementation of this solution.

[0033] The specific process of the lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search proposed in this embodiment can be as follows: Figure 1 As shown, including:

[0034] Step 101, construct and pre-train a dual-branch lightweight backbone network, the dual-branch lightweight backbone network includes an RGB backbone network and a deep backbone network, each backbone network is divided into five layers, including a first layer and a second layer for fast downsampling and shallow feature extraction, and a third layer, a fourth layer and a fifth layer with a residual structure for high-level semantic feature extraction.

[0035] In the specific implementation, the server first needs to build and pre-train a dual-branch lightweight backbone network, which includes an RGB backbone network and a deep backbone network. Each backbone network is divided into five layers, including the first and second layers for fast downsampling and shallow feature extraction, and the third, fourth and fifth layers with residual structures for high-level semantic feature extraction. When training the dual-branch lightweight backbone network, it is necessary to iteratively train the two branches based on the RGB sample images and the deep sample images respectively until convergence.

[0036] In one example, the structure of a dual-branch lightweight backbone network can be as follows Figure 2 As shown in the figure, the first and second layers are both ordinary convolution layers, each layer contains two 3×3 convolutions, the width and height of the output of the first and second layers are half of the input, and the number of channels of the output of the first and second layers is twice the input. Assuming that the size of the depth image input to the deep backbone network is w×h×c, w is the width, h is the height, and c is the number of channels, then the size of the feature map output by the second layer of the deep backbone network is w / 4×h / 4×4c. The first and second layers do not use residual structures because the resolution of the feature map is relatively large in the early stage of the backbone network. If a residual structure is used, it will inevitably bring a greater computational burden, thereby affecting the speed of the model.

[0037] In one example, the structure of a dual-branch lightweight backbone network can be as follows Figure 2 As shown, the third layer, the fourth layer and the fifth layer are all a lightweight module, and the lightweight module is formed by stacking a first sub-network and two second sub-networks. The network structure of the first sub-network and the second sub-network can be as follows Figure 3 As shown, the first subnetwork is Figure 3 GElayerS1 in the second subnetwork is Figure 3In GElayerS2, the first subnetwork includes a regular branch and a residual branch. The regular branch of the first subnetwork consists of 3×3 convolution, 3×3 depth-separable convolution with a stride of 2, 3×3 depth-separable convolution and 1×1 convolution. The residual branch of the first subnetwork consists of 3×3 depth-separable convolution with a stride of 2 and 1×1 convolution. The output of the first subnetwork is the pixel-by-pixel addition of the output of the regular branch of the first subnetwork and the output of the residual branch of the first subnetwork. The second subnetwork consists of 3×3 convolution, 3×3 depth-separable convolution and 1×1 convolution. The output of the second subnetwork is the pixel-by-pixel addition of the output of the 1×1 convolution of the second subnetwork and the input of the second subnetwork.

[0038] In one example, for the RGB backbone network, the feature map output by the first layer is The size of the second layer is w / 2×h / 2×2c, and the feature map output by the second layer The size of the third layer is w / 4×h / 4×4c, and the output feature map The size of the fourth layer is w / 8×h / 8×4c, and the feature map output by The size of the fifth layer is w / 16×h / 16×4c, and the feature map output by the fifth layer The size of is w / 32×h / 32×4c. For the deep backbone network, the feature map output by the first layer is The size of the second layer is w / 2×h / 2×2c, and the feature map output by the second layer The size of the third layer is w / 4×h / 4×4c, and the output feature map The size of the fourth layer is w / 8×h / 8×4c, and the feature map output by The size of the fifth layer is w / 16×h / 16×4c, and the feature map output by the fifth layer The dimensions are w / 32×h / 32×4c.

[0039] It can be understood that through gradual downsampling and feature extraction, the backbone network can effectively extract feature information at different levels from the original input image and provide richer semantic information and contextual understanding capabilities for subsequent tasks. It should also be noted that the backbone network introduced in this embodiment is only an exemplary description, but is not limited to this, and can be replaced by other semantic segmentation backbone networks.

[0040] Step 102, construct a hierarchical unit-level search space suitable for RGB-D image semantic segmentation, and obtain a cross-modal multi-scale feature adaptive fusion network, which consists of two RGB top units, two RGB middle units, two RGB bottom units, two depth top units, two depth middle units, two depth bottom units and three dynamic fusion modules, wherein feature maps of the same resolution share the same unit-level search space, and feature maps of different resolutions have different unit-level search spaces.

[0041] In the specific implementation, in addition to the dual-branch lightweight backbone network, the server also needs to build a hierarchical unit-level search space suitable for RGB-D image semantic segmentation. Feature maps of the same resolution share the same unit-level search space, and feature maps of different resolutions have different unit-level search spaces. Based on these unit-level search spaces, the server obtains a cross-modal multi-scale feature adaptive fusion network. Figure 2 As shown, the cross-modal multi-scale feature adaptive fusion network consists of two RGB top units ( Figure 2 RGB top cell in the RGB, two RGB middle cells ( Figure 2 RGB mid cell), two RGB bottom cells ( Figure 2 RGB bot cell in ), two depth top cells ( Figure 2 The depth top cell in the , two depth middle cells ( Figure 2 depth mid cell), two depth bottom cells ( Figure 2 The depth bot cell in Figure 2 This design can perform appropriate feature extraction on feature maps of different resolutions.

[0042] In an example, Figure 2 As shown, in the cross-modal multi-scale feature adaptive fusion network, the input of the second RGB top unit is the output of the first RGB top unit and the output of the first RGB intermediate unit, and the output of the second RGB top unit is recorded as , the input of the second RGB intermediate unit is the output of the first RGB intermediate unit and the output of the first RGB bottom unit, and the output of the second RGB intermediate unit is recorded as , the input of the second RGB bottom unit is the output of the first RGB top unit and the output of the first RGB bottom unit, and the output of the second RGB bottom unit is recorded as The input of the second depth top unit is the output of the first depth top unit and the output of the first depth middle unit. The output of the second depth top unit is recorded as , the input of the second depth intermediate unit is the output of the first depth intermediate unit and the output of the first depth bottom unit, and the output of the second depth intermediate unit is recorded as , the input of the second depth bottom unit is the output of the first depth top unit and the output of the first depth bottom unit, and the output of the second depth bottom unit is recorded as The input of the third dynamic fusion module is the output of the second RGB bottom unit , the output of the second depth bottom unit , and the output of the second RGB bottom unit and the output of the second depth bottom unit The result of element-wise addition of , the output of the third dynamic fusion module is recorded as , the input of the second dynamic fusion module is the output of the second RGB intermediate unit , the output of the second deep intermediate unit , and the output of the third dynamic fusion module , the output of the second dynamic fusion module is recorded as , the input of the first dynamic fusion module is the output of the second RGB top unit , the output of the top unit at the second depth , and the output of the second dynamic fusion module .

[0043] Step 103, after 3×3 convolution processing, the RGB image and the depth image in the multimodal input image are input into the RGB backbone network and the deep backbone network respectively, and the outputs of the third, fourth and fifth layers of the RGB backbone network and the outputs of the third, fourth and fifth layers of the deep backbone network are input into the cross-modal multi-scale feature adaptive fusion network for search, and optimized using the gradient descent optimization algorithm. After decoding, the optimal cross-modal multi-scale feature adaptive fusion network is obtained.

[0044] In the specific implementation, after the server completes the construction of the cross-modal multi-scale feature adaptive fusion network, it needs to optimize the cross-modal multi-scale feature adaptive fusion network. The server processes the RGB image and depth image in the multimodal input image through 3×3 convolution, and inputs them into the RGB backbone network and the deep backbone network respectively. The outputs of the third, fourth and fifth layers of the RGB backbone network, as well as the outputs of the third, fourth and fifth layers of the deep backbone network, are input into the cross-modal multi-scale feature adaptive fusion network for search, and are optimized using the gradient descent optimization algorithm. After decoding, the optimal cross-modal multi-scale feature adaptive fusion network is obtained.

[0045] In one example, the server sends the output of the third layer of the RGB backbone network And the output of the fourth layer , as the input of the first RGB top unit, the output of the fourth layer of the RGB backbone network And the output of the fifth layer , as the input of the first RGB intermediate unit, the output of the third layer of the RGB backbone network And the output of the fifth layer , as the input of the first RGB bottom unit, the output of the third layer of the deep backbone network And the output of the fourth layer , as the input of the first deep top unit, the output of the fourth layer of the deep backbone network And the output of the fifth layer , as the input of the first deep intermediate unit, the output of the third layer of the deep backbone network And the output of the fifth layer , as the input of the first depth bottom unit, thus searching.

[0046] In an example, the structures of the RGB top unit, RGB middle unit and RGB bottom unit in the cross-modal multi-scale feature adaptive fusion network are different, and the structures of the depth top unit, depth middle unit and depth bottom unit are different. Considering that the cross-modal multi-scale feature adaptive fusion network is a real-time network, the number of nodes in each unit should not be too many. Therefore, each unit is designed to consist of four nodes, including two input nodes, one middle node and one output node. The edge between the nodes is an operation in the operation set, and the search goal is to find the best operation on each edge.

[0047] In one example, the operation set consists of three types of operations, namely non-learning operations, standard convolutions and dilated convolutions, which together have a total of 11 operators. Non-learning operations include global average pooling, global maximum pooling and skip connections. Global average pooling includes 3×3 global average pooling and 5×5 global average pooling. Global maximum pooling includes 3×3 global maximum pooling and 5×5 global maximum pooling. Standard convolution includes 3×3 standard convolution, 5×5 standard convolution and 7×7 standard convolution. Dilated convolution includes dilated convolution with a dilation rate of 2, dilated convolution with a dilation rate of 4 and dilated convolution with a dilation rate of 8. Dilated convolution can well expand the field of view of the convolution kernel, improve the efficiency and accuracy of segmentation, and is very helpful for obtaining multi-scale information. It has a wide range of applications in the field of semantic segmentation. It is worth mentioning that both standard convolution and dilated convolution use deep separation technology, that is, decomposing the standard convolution into deep convolution and point convolution, which can improve the efficiency of convolution.

[0048] In an example, Figure 4 As shown in, for any unit in the cross-modal multi-scale feature adaptive fusion network, let its two input nodes be and The intermediate node is , the output node is , and It is also used to represent the output of two input nodes. It is also used to represent the output of the intermediate node. Also used to represent the output of an output node. Output and input nodes The output of each operation is respectively After the operations in the process are processed, they are transmitted to the intermediate nodes. , intermediate node The outputs of the two input nodes are summed and activated by the Relu activation function. The output of After processing, it is transferred to the output node .

[0049] In one example, the output node The output is expressed by the formula:

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] in, For operation collection The total number of operations in Represents a set of operations The operations, express The set of operations on this edge The The weighting factor of the operation, express The set of operations on this edge The The normalized weight coefficients of the operations, express The set of operations on this edge The The weighting factor of the operation, express The set of operations on this edge The The normalized weight coefficients of the operations, express The set of operations on this edge The The weighting factor of the operation, express The set of operations on this edge The The normalized weight coefficients of the operations.

[0055] It is worth noting that Composed of structure vector , structure vector Designed for the top unit, the middle unit and the bottom unit each have a corresponding structure vector and structure vector . Structure vector , structure vector and structure vector The discrete search space becomes continuous, so that the corresponding vector can be optimized using the gradient descent optimization algorithm, and finally the optimal network structure is obtained. After decoding, the optimal cross-modal multi-scale feature adaptive fusion network can be obtained.

[0056] In step 104, the outputs of the third, fourth and fifth layers of the RGB backbone network and the outputs of the third, fourth and fifth layers of the deep backbone network are input into the optimal cross-modal multi-scale feature adaptive fusion network for retraining to obtain the final cross-modal multi-scale feature adaptive fusion network.

[0057] In the specific implementation, after the server obtains the optimal cross-modal multi-scale feature adaptive fusion network, it can input the outputs of the third, fourth and fifth layers of the RGB backbone network and the outputs of the third, fourth and fifth layers of the deep backbone network into the optimal cross-modal multi-scale feature adaptive fusion network for retraining to obtain the final cross-modal multi-scale feature adaptive fusion network. The final cross-modal multi-scale feature adaptive fusion network can perform semantic segmentation tasks.

[0058] In an example, the working principle of the dynamic fusion module is introduced by taking the third dynamic fusion module as an example. Assume that the three inputs of the third dynamic fusion module are , and , and and the output of the second RGB bottom unit and the output of the second depth bottom unit, respectively, is the result of element-by-element addition of the output of the second RGB bottom unit and the output of the second depth bottom unit. The structure of the third dynamic fusion module is as follows Figure 5 As shown, the third dynamic fusion module consists of an RGB processing subnetwork, a depth processing subnetwork, a fusion subnetwork, and a projection subnetwork.

[0059] The RGB processing subnetwork consists of the first RGB processing branch, the second RGB processing branch and the first reshaping unit. The depth processing subnetwork consists of the first depth processing branch, the second depth processing branch and the second reshaping unit. The first RGB processing branch and the first depth processing branch are both a 1×1 convolutional layer. The second RGB processing branch and the second depth processing branch are both composed of two consecutive 3×3 convolutional layers. Each 3×3 convolutional layer is followed by a LeakRelu activation function layer.

[0060] Enter the first RGB processing branch, and After channel splicing, the second RGB processing branch is entered. The output of the first RGB processing branch and the output of the second RGB processing branch are added element by element and reshaped into the first feature by the first reshaping unit. Enter the first depth processing branch, and After channel splicing, the second deep processing branch is entered. The output of the first deep processing branch is added element by element with the output of the second deep processing branch, and then reshaped into the second feature by the second reshaping unit. Suppose the feature subspace of the first feature is , the characteristic subspace of the second feature is , Contains basis vectors, Also contains basis vectors, , .

[0061] The specific structure of the fusion sub-network can be as follows Figure 6 As shown, a gating unit is set in the fusion subnetwork, and four control operations are set in the gating unit: pixel-by-pixel addition, pixel-by-pixel subtraction, channel splicing and Hadamard product. The fusion subnetwork fuses the feature subspace corresponding to the first feature and the feature subspace corresponding to the second feature based on the gating unit to obtain the final feature subspace. The final feature subspace is recorded as , It also contains basis vectors, .

[0062] The projection subnetwork transforms Project it into the final feature subspace for reconstruction to obtain the reconstructed features As the output of the third dynamic fusion module. This process can be achieved by the following formula:

[0063] ;

[0064] ;

[0065] in, Indicates the final feature subspace, with the upper right corner represents the transpose operation. is an intermediate representation that has no actual meaning. represents the RGB input of the third dynamic fusion module, Represents the reconstruction feature, which is the output of the third dynamic fusion module.

[0066] It should be noted that the structure and working principle of the second dynamic fusion module and the first dynamic fusion module are the same as those of the third dynamic fusion module, except that the input is different, which will not be repeated here.

[0067] In an example, the operation of the fusion sub-network can be expressed by the following formula:

[0068] ;

[0069] ;

[0070] in, Indicates the gate control unit controls the operations, represents the desired one-hot generated by the gating unit The first dimension in the decision vector elements, Represents the first feature, Represents the second feature, Then represents the output of the fusion sub-network. Since the current gating unit provides discrete decisions, the relationship between branch selection and gating unit is not directly differentiable. Therefore, it is necessary to introduce the Gumbel-softmax function to reparameterize it and make the search space continuous.

[0071] Step 105, after upsampling the output of the third dynamic fusion module of the final cross-modal multi-scale feature adaptive fusion network, it is spliced ​​with the output of the second dynamic fusion module in the channel dimension, and then the features obtained after splicing are upsampled and spliced ​​with the output of the first dynamic fusion module in the channel dimension, and then after upsampling and a 3×3 convolution process, the semantic segmentation result of the multimodal input image is obtained.

[0072] In the specific implementation, Figure 2 As shown in the figure, the output of the third dynamic fusion module of the final cross-modal multi-scale feature adaptive fusion network is upsampled and then concatenated with the output of the second dynamic fusion module in the channel dimension. The concatenated features are then upsampled and concatenated with the output of the first dynamic fusion module in the channel dimension. After upsampling and a 3×3 convolution, the semantic segmentation result of the multimodal input image can be obtained.

[0073] In this embodiment, a dual-branch lightweight backbone network is constructed and pre-trained. The RGB backbone network in the dual-branch lightweight backbone network is used to extract features from RGB images, and the deep backbone network is used to extract features from deep images, which can fully exploit shallow features and high-level semantic features in multimodal images. A hierarchical unit-level search space suitable for semantic segmentation of RGB-D images is constructed to obtain a cross-modal multi-scale feature adaptive fusion network. Feature maps of the same resolution share the same unit-level search space, and feature maps of different resolutions have different unit-level search spaces, thereby performing appropriate feature extraction on feature maps of different resolutions. After search optimization and final training, the cross-modal multi-scale feature adaptive fusion network can be used for semantic segmentation tasks, which enhances the robustness of semantic segmentation of RGB-D real-time images, thereby achieving a good balance between semantic segmentation accuracy and semantic segmentation speed, and providing reliable data support for applications based on semantic segmentation.

[0074] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0075] In one example, in order to verify the effectiveness of a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search proposed in this application (hereinafter referred to as this method), we used the public dataset NYU-Depth v2 to train and test the network and compared it with other methods. NYU-Depth v2 contains 1449 RGB images and aligned depth images. The dataset is captured by Microsoft Kinect from 464 different scenes. NYU-Depth v2 has three annotations of different granularity, and we used dense annotations with 40 categories. The standard training set contains 795 images and the test set contains 654 images. This method achieved an average intersection-over-union ratio of 53.2% on the NYU-Depth v2 test set, and a speed of 67.2 frames per second on the GTX3090. While ensuring segmentation accuracy, it achieved a high inference speed, which can better meet the real-time requirements and outperform the performance of most models. Compared with the classic lightweight RGB-D semantic segmentation models ESANet-R34 and EMSANet, this method has higher advantages in segmentation accuracy and inference speed. When the backbone network is replaced with segformer, the average intersection-over-union ratio reaches 58.0% on the NYU-Depth v2 test set, and the speed reaches 18.6 frames per second on GTX3090. When the backbone network with a larger number of parameters is replaced, the speed of the model is reduced to a certain extent, but in exchange for a greater improvement in accuracy. However, compared with the RGB-D semantic segmentation model CMX using the same segformer backbone network, this method has advantages in both accuracy and speed, which reflects the effectiveness of this method.

[0076] Another embodiment of the present application proposes a lightweight RGB-D image semantic segmentation system based on hierarchical shared architecture search. The following is a specific description of the details of the lightweight RGB-D image semantic segmentation system based on hierarchical shared architecture search proposed in this embodiment. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this example. Figure 7 This is a structural diagram of a lightweight RGB-D image semantic segmentation system based on hierarchical shared architecture search proposed in this embodiment, including: a dual-branch lightweight backbone network construction module 201, a cross-modal multi-scale feature adaptive fusion network construction module 202, an optimization module 203, a final training module 204 and an execution module 205.

[0077] The dual-branch lightweight backbone network construction module 201 is used to construct and pre-train a dual-branch lightweight backbone network, which includes an RGB backbone network and a deep backbone network. Each backbone network is divided into five layers, including a first layer and a second layer for fast downsampling and shallow feature extraction, and a third layer, a fourth layer and a fifth layer with a residual structure for high-level semantic feature extraction.

[0078] A cross-modal multi-scale feature adaptive fusion network construction module 202 is used to construct a hierarchical unit-level search space suitable for RGB-D image semantic segmentation, and obtain a cross-modal multi-scale feature adaptive fusion network. The cross-modal multi-scale feature adaptive fusion network consists of two RGB top units, two RGB middle units, two RGB bottom units, two depth top units, two depth middle units, two depth bottom units and three dynamic fusion modules, wherein feature maps of the same resolution share the same unit-level search space, and feature maps of different resolutions have different unit-level search spaces.

[0079] The optimization module 203 is used to input the RGB image and the depth image in the multimodal input image into the RGB backbone network and the deep backbone network respectively after 3×3 convolution processing, and input the outputs of the third layer, fourth layer and fifth layer of the RGB backbone network, as well as the outputs of the third layer, fourth layer and fifth layer of the deep backbone network into the cross-modal multi-scale feature adaptive fusion network for search, and optimize them using the gradient descent optimization algorithm. After decoding, the optimal cross-modal multi-scale feature adaptive fusion network is obtained.

[0080] The final training module 204 is used to input the outputs of the third, fourth and fifth layers of the RGB backbone network and the outputs of the third, fourth and fifth layers of the deep backbone network into the optimal cross-modal multi-scale feature adaptive fusion network for retraining to obtain the final cross-modal multi-scale feature adaptive fusion network.

[0081] Execution module 205 is used to upsample the output of the third dynamic fusion module of the final cross-modal multi-scale feature adaptive fusion network, and then splice it with the output of the second dynamic fusion module in the channel dimension. Then, the features obtained after splicing are upsampled and then spliced ​​with the output of the first dynamic fusion module in the channel dimension. After upsampling and a 3×3 convolution process, the semantic segmentation result of the multimodal input image is obtained.

[0082] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0083] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above method embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above method embodiment.

[0084] Another embodiment of the present application provides an electronic device, whose specific structure can be as follows: Figure 8 As shown, it includes: at least one processor 301; and a memory 302 that is communicatively connected to the at least one processor 301; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 so that the at least one processor 301 can execute a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search as described in the above method embodiment.

[0085] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators and power management circuits together, which are all well known in the art, so this article will not describe them further. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor.

[0086] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0087] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search as described in the above method embodiment.

[0088] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0089] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search, characterized in that: The method comprises: Construct and pre-train a dual-branch lightweight backbone network, which includes an RGB backbone network and a deep backbone network. Each backbone network is divided into five layers, including the first and second layers for fast downsampling and shallow feature extraction, and the third, fourth and fifth layers with a residual structure for high-level semantic feature extraction; A hierarchical unit-level search space suitable for RGB-D image semantic segmentation is constructed to obtain a cross-modal multi-scale feature adaptive fusion network, which consists of two RGB top units, two RGB middle units, two RGB bottom units, two deep top units, two deep middle units, two deep bottom units and three dynamic fusion modules; among them, feature maps of the same resolution share the same unit-level search space, and feature maps of different resolutions have different unit-level search spaces; After 3×3 convolution processing, the RGB image and the depth image in the multimodal input image are input into the RGB backbone network and the depth backbone network respectively. The outputs of the third, fourth and fifth layers of the RGB backbone network and the third, fourth and fifth layers of the depth backbone network are input into the cross-modal multi-scale feature adaptive fusion network for search and optimized using the gradient descent optimization algorithm. After decoding, the optimal cross-modal multi-scale feature adaptive fusion network is obtained. The outputs of the third, fourth and fifth layers of the RGB backbone network and the outputs of the third, fourth and fifth layers of the deep backbone network are input into the optimal cross-modal multi-scale feature adaptive fusion network for retraining to obtain the final cross-modal multi-scale feature adaptive fusion network; The output of the third dynamic fusion module of the final cross-modal multi-scale feature adaptive fusion network is upsampled and concatenated with the output of the second dynamic fusion module in the channel dimension. The concatenated features are then upsampled and concatenated with the output of the first dynamic fusion module in the channel dimension. After upsampling and a 3×3 convolution, the semantic segmentation result of the multimodal input image is obtained. The structures of the RGB top unit, RGB middle unit and RGB bottom unit in the cross-modal multi-scale feature adaptive fusion network are different. The structures of the depth top unit, depth middle unit and depth bottom unit are different. Each unit consists of four nodes, including two input nodes, one middle node and one output node. The edge between the nodes is an operation in the operation set. The search goal is to find the best operation on each edge. The operation set consists of three types of operations, namely non-learning operations, standard convolutions and dilated convolutions. Non-learning operations include global average pooling, global maximum pooling and skip connections. Global average pooling includes 3×3 global average pooling and 5×5 global average pooling. Global maximum pooling includes 3×3 global maximum pooling and 5×5 global maximum pooling. Standard convolution includes 3×3 standard convolution, 5×5 standard convolution and 7×7 standard convolution. Dilated convolution includes dilated convolution with a dilation rate of 2, dilated convolution with a dilation rate of 4 and dilated convolution with a dilation rate of 8. For the third dynamic fusion module, let its three inputs be , and , and and the output of the second RGB bottom unit and the output of the second depth bottom unit, respectively, is the result of element-wise addition of the output of the second RGB bottom unit and the output of the second depth bottom unit; The third dynamic fusion module consists of an RGB processing subnetwork, a depth processing subnetwork, a fusion subnetwork and a projection subnetwork. The RGB processing subnetwork consists of a first RGB processing branch, a second RGB processing branch and a first reshaping unit. The depth processing subnetwork consists of a first depth processing branch, a second depth processing branch and a second reshaping unit. The first RGB processing branch and the first depth processing branch are both a 1×1 convolutional layer. The second RGB processing branch and the second depth processing branch are both composed of two consecutive 3×3 convolutional layers. Each 3×3 convolutional layer is followed by a LeakRelu activation function layer. Enter the first RGB processing branch, and After channel splicing, the second RGB processing branch is entered. The output of the first RGB processing branch and the output of the second RGB processing branch are added element by element and reshaped into the first feature by the first reshaping unit. Enter the first depth processing branch, and After channel splicing, the second deep processing branch is entered, and the output of the first deep processing branch and the output of the second deep processing branch are added element by element, and then reshaped into a second feature by a second reshaping unit; The fusion sub-network is provided with a gating unit, and the gating unit is provided with four control operations: pixel-by-pixel addition, pixel-by-pixel subtraction, channel splicing and Hadamard product. The fusion sub-network fuses the feature subspace corresponding to the first feature and the feature subspace corresponding to the second feature based on the gating unit to obtain the final feature subspace; The projection subnetwork transforms Project it into the final feature subspace for reconstruction to obtain the reconstructed features As the output of the third dynamic fusion module; The structures and working principles of the second dynamic fusion module and the first dynamic fusion module are the same as those of the third dynamic fusion module, except that the inputs are different.

2. According to claim 1, a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search is characterized in that: The first and second layers of the backbone network are both ordinary convolutional layers. Each layer contains two 3×3 convolutions. The width and height of the output of the first and second layers of the backbone network are half of the input, and the number of channels of the output of the first and second layers of the backbone network are twice that of the input. The third, fourth and fifth layers of the backbone network are all lightweight modules, which are composed of a first sub-network and two second sub-networks stacked together. The first sub-network includes a regular branch and a residual branch. The regular branch of the first sub-network consists of 3×3 convolution, 3×3 depth-separable convolution with a step size of 2, 3×3 depth-separable convolution and 1×1 convolution. The residual branch of the first sub-network consists of 3×3 depth-separable convolution with a step size of 2 and 1×1 convolution. The output of the first sub-network is the pixel-by-pixel addition of the output of the regular branch of the first sub-network and the output of the residual branch of the first sub-network. The second sub-network consists of 3×3 convolution, 3×3 depth-separable convolution and 1×1 convolution. The output of the second sub-network is the pixel-by-pixel addition of the output of the 1×1 convolution of the second sub-network and the input of the second sub-network.

3. The lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search according to claim 2, characterized in that: In the cross-modal multi-scale feature adaptive fusion network, the input of the second RGB top unit is the output of the first RGB top unit and the output of the first RGB middle unit, the input of the second RGB middle unit is the output of the first RGB middle unit and the output of the first RGB bottom unit, the input of the second RGB bottom unit is the output of the first RGB top unit and the output of the first RGB bottom unit, the input of the second depth top unit is the output of the first depth top unit and the output of the first depth middle unit, the input of the second depth middle unit is the output of the first depth middle unit and the output of the first depth bottom unit, the input of the second depth bottom unit is the output of the first depth top unit and the output of the first depth bottom unit, the input of the third dynamic fusion module is the output of the second RGB bottom unit, the output of the second depth bottom unit, and the result of element-by-element addition of the output of the second RGB bottom unit and the output of the second depth bottom unit, the input of the second dynamic fusion module is the output of the second RGB middle unit, the output of the second depth middle unit, and the output of the third dynamic fusion module, and the input of the first dynamic fusion module is the output of the second RGB top unit, the output of the second depth top unit, and the output of the second dynamic fusion module.

4. The lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search according to claim 3, characterized in that: The outputs of the third, fourth and fifth layers of the RGB backbone network and the third, fourth and fifth layers of the deep backbone network are input into the cross-modal multi-scale feature adaptive fusion network for search, including: The outputs of the third and fourth layers of the RGB backbone network are used as the input of the first RGB top unit, the outputs of the fourth and fifth layers of the RGB backbone network are used as the input of the first RGB middle unit, the outputs of the third and fifth layers of the RGB backbone network are used as the input of the first RGB bottom unit, the outputs of the third and fourth layers of the deep backbone network are used as the input of the first deep top unit, the outputs of the fourth and fifth layers of the deep backbone network are used as the input of the first deep middle unit, and the outputs of the third and fifth layers of the deep backbone network are used as the input of the first deep bottom unit to perform the search.

5. The lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search according to claim 1, characterized in that: For any unit in the cross-modal multi-scale feature adaptive fusion network, let its two input nodes be and The intermediate node is , the output node is , and It is also used to represent the output of two input nodes. It is also used to represent the output of the intermediate node. It is also used to represent the output of an output node and the input node Output and input nodes The output of each operation is respectively After the operations in the process are processed, they are transmitted to the intermediate nodes. , intermediate node The outputs of the two input nodes are summed and activated by the Relu activation function. The output of After processing, it is transferred to the output node , output node The output is expressed by the formula: ； ； ； ； in, For operation collection The total number of operations in Represents a set of operations The operations, express The set of operations on this edge The The weighting factor of the operation, express The set of operations on this edge The The normalized weight coefficients of the operations, express The set of operations on this edge The The weighting factor of the operation, express The set of operations on this edge The The normalized weight coefficients of the operations, express The set of operations on this edge The The weighting factor of the operation, express The set of operations on this edge The The normalized weight coefficient of the operation.

6. A lightweight RGB-D image semantic segmentation system based on hierarchical shared architecture search, characterized in that: The system is composed of a dual-branch lightweight backbone network construction module, a cross-modal multi-scale feature adaptive fusion network construction module, an optimization module, a final training module and an execution module; A dual-branch lightweight backbone network construction module is used to construct and pre-train a dual-branch lightweight backbone network. The dual-branch lightweight backbone network includes an RGB backbone network and a deep backbone network. Each backbone network is divided into five layers, including the first and second layers for fast downsampling and shallow feature extraction, and the third, fourth and fifth layers with a residual structure for high-level semantic feature extraction; A cross-modal multi-scale feature adaptive fusion network construction module is used to construct a hierarchical unit-level search space suitable for RGB-D image semantic segmentation, and obtain a cross-modal multi-scale feature adaptive fusion network. The cross-modal multi-scale feature adaptive fusion network consists of two RGB top units, two RGB middle units, two RGB bottom units, two depth top units, two depth middle units, two depth bottom units and three dynamic fusion modules, wherein feature maps of the same resolution share the same unit-level search space, and feature maps of different resolutions have different unit-level search spaces; The optimization module is used to process the RGB image and the depth image in the multimodal input image through 3×3 convolution, and then input them into the RGB backbone network and the depth backbone network respectively, and input the outputs of the third, fourth and fifth layers of the RGB backbone network and the outputs of the third, fourth and fifth layers of the depth backbone network into the cross-modal multi-scale feature adaptive fusion network for search, and optimize them using the gradient descent optimization algorithm, and after decoding, obtain the optimal cross-modal multi-scale feature adaptive fusion network; The final training module is used to input the outputs of the third, fourth and fifth layers of the RGB backbone network and the outputs of the third, fourth and fifth layers of the deep backbone network into the optimal cross-modal multi-scale feature adaptive fusion network for retraining to obtain the final cross-modal multi-scale feature adaptive fusion network; An execution module is used to upsample the output of the third dynamic fusion module of the final cross-modal multi-scale feature adaptive fusion network, and then splice it with the output of the second dynamic fusion module in the channel dimension. Then, the features obtained after splicing are upsampled and then spliced ​​with the output of the first dynamic fusion module in the channel dimension. After upsampling and a 3×3 convolution process, the semantic segmentation result of the multimodal input image is obtained. The structures of the RGB top unit, RGB middle unit and RGB bottom unit in the cross-modal multi-scale feature adaptive fusion network are different. The structures of the depth top unit, depth middle unit and depth bottom unit are different. Each unit consists of four nodes, including two input nodes, one middle node and one output node. The edge between the nodes is an operation in the operation set. The search goal is to find the best operation on each edge. The operation set consists of three types of operations, namely non-learning operations, standard convolutions and dilated convolutions. Non-learning operations include global average pooling, global maximum pooling and skip connections. Global average pooling includes 3×3 global average pooling and 5×5 global average pooling. Global maximum pooling includes 3×3 global maximum pooling and 5×5 global maximum pooling. Standard convolution includes 3×3 standard convolution, 5×5 standard convolution and 7×7 standard convolution. Dilated convolution includes dilated convolution with a dilation rate of 2, dilated convolution with a dilation rate of 4 and dilated convolution with a dilation rate of 8. For the third dynamic fusion module, let its three inputs be , and , and and the output of the second RGB bottom unit and the output of the second depth bottom unit, respectively, is the result of element-wise addition of the output of the second RGB bottom unit and the output of the second depth bottom unit; The third dynamic fusion module consists of an RGB processing subnetwork, a depth processing subnetwork, a fusion subnetwork and a projection subnetwork. The RGB processing subnetwork consists of a first RGB processing branch, a second RGB processing branch and a first reshaping unit. The depth processing subnetwork consists of a first depth processing branch, a second depth processing branch and a second reshaping unit. The first RGB processing branch and the first depth processing branch are both a 1×1 convolutional layer. The second RGB processing branch and the second depth processing branch are both composed of two consecutive 3×3 convolutional layers. Each 3×3 convolutional layer is followed by a LeakRelu activation function layer. Enter the first RGB processing branch, and After channel splicing, the second RGB processing branch is entered. The output of the first RGB processing branch and the output of the second RGB processing branch are added element by element and reshaped into the first feature by the first reshaping unit. Enter the first depth processing branch, and After channel splicing, the second deep processing branch is entered, and the output of the first deep processing branch and the output of the second deep processing branch are added element by element, and then reshaped into a second feature by a second reshaping unit; The fusion sub-network is provided with a gating unit, and the gating unit is provided with four control operations: pixel-by-pixel addition, pixel-by-pixel subtraction, channel splicing and Hadamard product. The fusion sub-network fuses the feature subspace corresponding to the first feature and the feature subspace corresponding to the second feature based on the gating unit to obtain the final feature subspace; The projection subnetwork transforms Project it into the final feature subspace for reconstruction to obtain the reconstructed features As the output of the third dynamic fusion module; The structures and working principles of the second dynamic fusion module and the first dynamic fusion module are the same as those of the third dynamic fusion module, except that the inputs are different.

7. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement a lightweight RGB-D image semantic segmentation method based on hierarchical shared architecture search as described in any one of claims 1 to 5.

Images