Intelligent driving semantic segmentation method and acceleration system adaptive to complex climate

Abstract

The application discloses an intelligent driving real-time semantic segmentation method and system suitable for complex climates, and aims to solve the problems of low segmentation accuracy, poor real-time performance and hardware deployment of existing semantic segmentation methods under complex climates (rain, fog, snow, night, etc.). The application constructs a hardware-friendly single-branch CNN semantic segmentation network, proposes a generation denoising-weighted distillation training strategy, guides the CNN student branch to learn the invariance features of complex climates through the Transformer teacher branch, improves the model robustness and segmentation performance, and simultaneously optimizes the segmentation boundary and relieves the training imbalance problem by combining a boundary enhancement label generation module. The hardware acceleration system of the application is based on an FPGA hardware platform, adopts a software and hardware collaborative design, realizes efficient and real-time inference of the algorithm through a flow water reuse calculation architecture and a preprocessing module. The application can meet the intelligent driving real-time environmental perception requirements by taking into account the accuracy and speed in complex climate scenes.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent driving environment perception, computer vision semantic segmentation and hardware acceleration technology. Specifically, it relates to an intelligent driving real-time semantic segmentation method adapted to complex climates, and an FPGA hardware acceleration system for deploying the inference network in this method. Background Technology

[0002] With the rapid development of autonomous driving technology, the requirements for environmental perception capabilities of intelligent driving systems are constantly increasing. Level 5 autonomous driving requires visual perception systems to be able to analyze input images in various complex scenarios and achieve accurate identification of targets such as roads, vehicles, and pedestrians. Semantic segmentation technology, as a core pixel-level classification method, directly determines the reliability of intelligent driving environmental perception and is a key support for ensuring vehicle driving safety. However, in real-world road scenarios, in addition to normal sunny weather, rain, fog, snow, and nighttime conditions pose severe challenges to the identification and segmentation of driving spaces and general obstacles based on RGB images due to factors such as illumination distortion and noise interference. At the same time, intelligent driving has extremely stringent real-time requirements for semantic segmentation, while the hardware resources of embedded in-vehicle devices are limited, further exacerbating the difficulty of deploying high-precision semantic segmentation algorithms.

[0003] Convolutional Neural Networks (CNNs) are currently one of the mainstream architectures for real-time semantic segmentation networks, with representative models including BiSeNet, DDRNet, PIDNet, and various lightweight models. However, most current CNN-based real-time semantic segmentation methods employ dual-branch or multi-branch structures, adding extra network branches to extract multi-scale information to improve segmentation accuracy. However, multi-branch structures introduce additional information redundancy and computational costs, leading to slower inference speeds. Furthermore, lightweight CNN networks still struggle to meet the safety requirements of autonomous driving in complex weather scenarios. Transformer-based semantic segmentation methods, with their self-attention mechanism, possess excellent long-range context extraction capabilities, effectively capturing the global correlation between targets and backgrounds in complex weather conditions, thus alleviating segmentation challenges caused by complex weather to some extent. However, the quadratic computational complexity of Transformers results in extremely high inference latency, making it difficult to adapt to the computational limitations of in-vehicle embedded devices and failing to meet the real-time requirements of autonomous driving.

[0004] In summary, a real-time semantic segmentation method and hardware acceleration system are needed for intelligent driving scenarios in complex weather conditions to meet the high-precision, real-time semantic segmentation computation requirements of in-vehicle edge computing. In complex weather driving scenarios, images are noisy and target contours are blurred, requiring networks to efficiently extract and utilize spatial and semantic features while possessing strong noise resistance. Single-branch CNN networks, as hardware-friendly networks, offer fast inference speeds and are suitable for deployment at in-vehicle edge computing. However, existing single-branch CNN networks struggle to fully learn long-range contextual features in complex weather scenarios, resulting in insufficient segmentation boundary accuracy, and lack effective training strategies to alleviate the training imbalance problem under different weather conditions. Therefore, it is necessary to improve the training strategy of single-branch CNN networks, design optimized auxiliary training methods such as generative denoising-weighted distillation and boundary enhancement, and guide the training of single-branch CNN networks by pre-training converged Transformer teacher branches. While maintaining the network's lightweight and hardware-friendly nature, this further improves the network's segmentation accuracy and robustness for complex weather conditions. At the same time, an FPGA hardware acceleration system that is deeply adapted to the optimized single-branch CNN network is designed to ensure efficient and fast semantic segmentation tasks under limited on-board hardware resources, thus addressing the pain points of existing technologies. Summary of the Invention

[0005] Purpose of the invention: To address the problems existing in the prior art, this invention provides a real-time semantic segmentation method and hardware acceleration system for intelligent driving that adapts to complex climates. This solves the problems of low segmentation accuracy and poor real-time performance of existing methods under complex climates, as well as hardware deployment issues, thereby meeting the environmental perception requirements of intelligent driving.

[0006] Technical solution: The present invention provides a real-time semantic segmentation method for intelligent driving adapted to complex climates, comprising the following steps:

[0007] Step S1: Based on the encoder-decoder architecture, construct the network structure of a single-branch CNN-type semantic segmentation network;

[0008] Step S2: Build a Transformer-type teacher branch for the encoder-decoder architecture. Train it to convergence on a complex climate dataset consisting of the normal climate dataset Cityscapes and the adverse climate dataset ACDC to obtain a converged Transformer teacher branch.

[0009] Step S3: Construct a boundary enhancement label generation module and a boundary detail loss function to alleviate the training imbalance problem between normal and adverse weather conditions;

[0010] Step S4: Based on the network structure of a single-branch CNN-type semantic segmentation network, a generation denoising-weighted distillation strategy is constructed. Gaussian noise is added to the intermediate features of the CNN student branch for denoising. Then, channel weighting is achieved through the channel attention mechanism. Combined with KL divergence loss, the CNN student branch learns the remote context features of the Transformer teacher branch.

[0011] Step S5: The constructed single-branch CNN-type semantic segmentation network is used as the student branch. Under the guidance of the Transformer teacher branch that converged in step S2, the boundary detail loss and the generation denoising-weighted distillation strategy are fused for joint training. The intelligent driving scene image under complex weather conditions is input into the trained and converged single-branch CNN-type semantic segmentation network, and the pixel-level semantic segmentation result is output to complete the real-time semantic segmentation.

[0012] Furthermore, in step S1, the encoder of the single-branch CNN-type semantic segmentation network adopts a ResNet-like residual backbone network. The intermediate features of the encoder are divided into four layers, labeled as layer 1 to layer 4. After adjusting the number of channels of the features of layer 4 through 1×1 convolution, bilinear upsampling is performed to the size of the features of layer 2, and channel splicing and fusion are performed. The fused features are sent to the decoder to output the segmentation result.

[0013] Furthermore, in step S2, the Transformer-type teacher branch adopts an encoder-decoder architecture adapted to a single-branch CNN-type semantic segmentation network. The encoder's backbone outputs four layers of intermediate features, labeled as layers 1 to 4. The features of layers 2 to 4 are bilinearly upsampled to the size of the features of layer 1, and then channel-by-channel splicing and fusion are performed. The fused features are then sent to the decoder to output the segmentation result.

[0014] Furthermore, in step S3, the boundary enhancement label generation module and the boundary detail loss function are specifically as follows: For the actual labels used in training, the boundary enhancement label generation module uses two-dimensional boundary convolution kernels, horizontal convolution kernels, and vertical convolution kernels with fixed parameters to acquire multi-scale detail information with different step lengths. After upsampling, dynamic weighted summation is performed through 1×1 convolution, and the result is converted into binary boundary enhancement labels according to the set threshold. The calculation formula is as follows:

[0015]

[0016] In the formula, This represents the boundary enhancement label in the i-th row and j-th column. This represents the feature in the i-th row and j-th column obtained by calculating and fusing two-dimensional boundary convolution kernels, horizontal convolution kernels, and vertical convolution kernels with fixed parameters;

[0017] The DICE loss and binary cross-entropy loss are calculated using the boundary enhancement labels and the second-layer features of the CNN branch, respectively. The boundary detail loss is obtained through the binary cross-entropy loss and the DICE loss. The calculation formula is as follows:

[0018]

[0019]

[0020] In the formula, H and W are the height and width of the second-layer feature and boundary enhancement label, respectively. It is a DICE loss. It is a binary cross-entropy loss. It is a loss of boundary details. The segmentation prediction is obtained from the output features of the second-layer encoder. For the generated boundary enhancement labels, For feature height, The feature width.

[0021] Furthermore, in step S4, the construction of the denoising-weighted distillation strategy specifically involves: adding the features of the 3rd and 4th layers of the CNN branch to Gaussian noise with a mean of 0 and a variance of 1, respectively; using a backbone feature alignment module to achieve denoising and channel dimension alignment, so that the number of channels of the features of the 3rd and 4th layers of the CNN branch is consistent with the number of channels of the 3rd and 4th layers of the teacher branch; then, using a channel attention mechanism to perform channel weighting on the features of the 3rd and 4th layers of the CNN branch, and calculating the KL divergence between the weighted features and the teacher features of the 3rd and 4th layers respectively, as the distillation loss.

[0022] Furthermore, the branch CNN semantic segmentation network is the core inference network of the semantic segmentation method; the Transformer teacher branch, the boundary enhancement label generation module, and the backbone feature alignment module in the generation denoising-weighted distillation strategy are all auxiliary modules in the network training stage and do not participate in real-time inference during hardware deployment.

[0023] This invention also discloses a hardware acceleration system for real-time semantic segmentation of intelligent driving that adapts to complex climates. Based on FPGA, the inference process of a single-branch CNN semantic segmentation network that can be deployed after training is implemented in the semantic segmentation method.

[0024] The hardware acceleration system consists of two parts: the PS and PL hardware accelerators. The PS is used to implement software-level control and post-processing, including the pre-storage of input images and weight parameters, the final post-processing of segmentation results, and communication with the PL.

[0025] The PL terminal implements a fully pipelined hardware circuit design, specifically adapted for the forward inference computation of the aforementioned CNN branch semantic segmentation network, including:

[0026] The AXI bus unit is used to realize bidirectional data transmission between the PS and PL ends. It receives the network model weights, feature data and network layer parameter information sent by the PS end, and sends back the 4th layer features and the final inference result of the PL end to the PS end.

[0027] The control unit is used to read the network layer information being calculated, decode it, and generate control and configuration information for each module to realize the reading and writing of feature maps and the reading of weight data, ensuring the smooth execution of data reception, calculation, and transmission tasks.

[0028] The caching unit includes a feature map caching module and a weight caching module, which are used to cache externally written feature map and weight data as well as intermediate features output by the parallel computing unit.

[0029] The preprocessing unit is used to generate the access addresses of the feature map and weight storage module, and to detect the padding region in real time. It provides the feature map window required for convolution calculation to the computing unit, while avoiding caching and reading of padding zero values, thus improving computing efficiency.

[0030] The pipelined computing unit is used to realize the pipelined computation of convolutional layer fusion between continuous network layers. The first stage uses 4 groups of 3×3 PE arrays with a depth of 16, and the second stage uses 32 groups of 3×3 PE arrays with a depth of 2. When the output features of the first-stage computing unit form a complete feature window, the convolutional computation of the second-stage computing unit can be started in parallel.

[0031] The residual computation unit uses four 1×1 PE arrays with a depth of 2. By reusing the input data of the first-level computation unit, it realizes the residual connection operation in the backbone of the CNN branch network.

[0032] Bilinear Upsampling Unit: Responsible for implementing bilinear interpolation upsampling operations on the output features of the fourth layer in the model. This computation unit adopts a three-level row buffer module design to realize pipelined computation of the upsampled data stream.

[0033] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0034] The present invention also discloses a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method of the present invention.

[0035] The present invention also discloses a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method of the present invention.

[0036] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0037] A generative denoising-weighted distillation training strategy is constructed, where the Transformer teacher branch guides the CNN student branch's learning. Gaussian noise doping and denoising processing are combined to enhance the model's adaptability to complex weather conditions. A boundary enhancement label generation module generates segmentation boundaries to assist training, alleviating the training imbalance problem and enabling the model to achieve accurate segmentation under both normal and complex weather conditions. The converged CNN branch achieves an mIoU of 79.93% on the Cityscapes dataset and 73.08% on the ACDC dataset, demonstrating highly competitive segmentation performance in driving scenarios with complex weather conditions compared to existing real-time semantic segmentation techniques. On the GTX 3090 platform, the CNN branch achieves an inference speed of 105.2 FPS when processing input images with a resolution of 1536×768, demonstrating good real-time performance.

[0038] The CNN branch network adopts a simple and efficient single-branch architecture, which is more hardware-friendly and easier to deploy on edge devices. It also employs a two-stage pipelined multiplexing architecture to implement a layer fusion deployment strategy, and combined with the design of the preprocessing unit, reduces computational complexity and storage requirements, decreasing reliance on hardware resources and enabling the network to run efficiently on resource-constrained edge devices. When deployed on the ZCU102 hardware platform, for an input image with a resolution of 768×384, an inference speed of 31.2 FPS using 1208 DSP resources can be achieved, meeting the real-time requirements of autonomous driving. Attached Figure Description

[0039] Figure 1 This is a structural diagram of a single-branch CNN-type semantic segmentation network and a Transformer-type teacher-branch semantic segmentation network provided according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of a noise reduction-weighted distillation method provided according to an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of a hardware accelerator implementation according to an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of a row buffer module and a register cache matrix provided according to an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of a continuous network interlayer fusion pipeline calculation method provided by an embodiment of the present invention. Detailed Implementation

[0044] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0045] The real-time semantic segmentation method for intelligent driving adapted to complex weather conditions provided in this invention performs the following steps S1-S5: by improving the training method of a single-branch CNN-type semantic segmentation network, and using intelligent driving scene images under complex weather conditions as input, real-time semantic segmentation is completed.

[0046] Step S1: Based on the encoder-decoder architecture, construct the basic network structure of a single-branch CNN-type semantic segmentation network;

[0047] The basic network structure of a single-branch CNN-type semantic segmentation network is referenced. Figure 1 The encoder employs a ResNet-like residual backbone network, consisting of four residual network modules. The intermediate features generated by these four modules are named as layer 1 to layer 4 intermediate features, from lowest to highest. Each residual network module is further composed of several basic residual blocks, with the four modules containing 2, 2, 4, and 5 basic residual blocks respectively. An original image of size (H, W, 3) is input to the residual network modules and rapidly downsampled using two 3×3 convolutions with a stride of 2, resulting in a feature size of (H / 4, W / 4, 32). The first residual module generates intermediate features of size (H / 4, W / 4, 32). Subsequent residual modules generate intermediate features with halved width and height and doubled channel count compared to the previous layer.

[0048] Since low-level features primarily extract spatial information from images, while high-level features focus on extracting semantic information, a 1×1 convolution is used to adjust the number of channels in the fourth-layer intermediate features of the encoder to 64, and bilinear upsampling is employed to change the feature size to (H / 8, W / 8). This is then combined with the second-layer intermediate features through channel concatenation, thus efficiently fusing spatial and semantic information. The fused features are then fed into the decoder to output the segmentation result.

[0049] Step S2: Use Segformer-b3 as the Transformer-type teacher branch semantic segmentation network described in Step S2. Train it in advance on a complex climate dataset consisting of the normal climate dataset Cityscapes and the adverse climate dataset ACDC until convergence. The mixed dataset contains images of five scenes: sunny, foggy, rainy, snowy, and nighttime, to ensure that the teacher branch can learn the semantic features under complex climate conditions.

[0050] The teacher branch network has an encoder-decoder architecture adapted to the aforementioned single-branch CNN-type semantic segmentation network. Its encoder backbone outputs four layers of intermediate features, labeled layers 1 to 4. The size of the first-layer intermediate feature is (H / 4, W / 4, 64). Subsequent intermediate features are generated with half the size and double the number of channels compared to the previous layer. The features from layers 2 to 4 are bilinearly upsampled to the size of the first-layer features, and then concatenated and fused. The fused features are then fed into the decoder to output the segmentation result.

[0051] Step S3: Construct a boundary enhancement label generation module and a boundary detail loss function to alleviate the training imbalance problem between normal and complex weather conditions;

[0052] The principle of the boundary enhancement label generation module and boundary detail loss mentioned in step S3 is as follows:

[0053] For the actual labels used during training, the boundary enhancement label generation module uses two-dimensional boundary convolution kernels, horizontal convolution kernels, and vertical convolution kernels with fixed parameters, and strides set to 1, 2, and 3 respectively, to obtain multi-scale spatial boundary detail information.

[0054] The boundary convolution kernel, horizontal convolution kernel, and vertical convolution kernel are shown below:

[0055] The generated multi-scale boundary enhancement labels are upsampled and then dynamically weighted using 1×1 convolution. Based on a set threshold, they are converted into binary boundary enhancement labels. The calculation formula is as follows:

[0056]

[0057] Using the boundary enhancement labels and the features from the second layer of the CNN branch, the binary cross-entropy loss and DICE loss are calculated separately. The boundary detail loss is obtained by weighted summation of the two, as shown in the following formula:

[0058]

[0059]

[0060] In the formula, H and W are the height and width of the second-layer feature and boundary enhancement label, respectively. It is a DICE loss. It is a binary cross-entropy loss. It is a loss of boundary details.

[0061] Step S4: Based on the basic network structure of a single-branch CNN-type semantic segmentation network, a generation denoising-weighted distillation strategy is constructed. Gaussian noise is added to the intermediate features of the CNN student branch for denoising. Then, channel weighting is achieved through the channel attention mechanism. Combined with KL divergence loss, the CNN student branch learns the remote context features of the Transformer teacher branch.

[0062] The generative denoising-weighted distillation strategy constructed in step S4 is as follows:

[0063] Reference Figure 1 and Figure 2 The features of layers 3 and 4 of the CNN branch are added to Gaussian noise with a mean of 0 and a variance of 1, respectively, with the same noise value added at the same position in different channels. Then, an MLP containing two 1×1 convolutional layers is used as the backbone feature alignment module to implement the denoising process, thereby enhancing the robustness of the model. Simultaneously, the backbone feature alignment module can perform channel dimension alignment, ensuring that the number of channels in the features of layers 3 and 4 of the CNN branch is consistent with the number of channels in layers 3 and 4 of the Transformer teacher branch.

[0064] Then, through the SE channel attention mechanism, the features of the 3rd and 4th layers of the CNN branch after channel dimension alignment are channel-weighted respectively. The KL divergence of the weighted student features and the teacher features of the 3rd and 4th layers is calculated as the distillation loss value to guide the CNN student branch to learn the remote context features of the teacher branch.

[0065] Step S5: The constructed single-branch CNN-type semantic segmentation network is used as the student branch. Under the guidance of the Transformer teacher branch that has been trained and converged in step S2, the boundary detail loss and the generation denoising-weighted distillation strategy are fused for joint training. The intelligent driving scene image under complex weather conditions is input into the trained and converged single-branch CNN network, and the pixel-level semantic segmentation result is output to complete the real-time semantic segmentation.

[0066] Training was implemented using the PyTorch framework. The main deep learning environment setup was CUDA 12.4 and PyTorch 2.1.0. The initial learning rate for the teacher model was 0.00006, with weight decay of 0.01, using the AdamW optimizer, a batch size of 4, and 160k iterations until model convergence. During the distillation training of the CNN branch, the initial learning rate was 0.00004, with weight decay of 0.0125, using the AdamW optimizer, a batch size of 12, and 160k iterations until model convergence. The Cityscapes dataset (sunny weather city street view images) and the ACDC dataset (containing four scenes: fog, rain, snow, and night, with pixel-level panoramic annotations) were selected and randomly mixed to obtain a Hybrid complex climate dataset, which was then divided into training and validation sets in an 8:2 ratio.

[0067] Reference Figure 3 In another aspect, this invention provides a hardware acceleration system for real-time semantic segmentation in intelligent driving that adapts to complex climates. Based on the Xilinx ZCU102 hardware platform, it adopts a hardware-software co-design and is specifically designed to realize real-time inference of the single-branch CNN-type semantic segmentation network trained in the above-mentioned semantic segmentation method. It includes a PS (central processing system) end and a PL (programmable logic unit) end, and the functions of each part are as follows:

[0068] PS end: Utilizes an ARM processor to handle software-level control and post-processing, including pre-storing the input image and network weight parameters in off-chip DRAM; sending start commands and network layer configuration parameters to PL end; receiving convolution calculation results from PL end; and completing the final pixel category mapping and segmentation result output.

[0069] PL (Programmer) side: Responsible for hardware circuit design and implementing forward inference computation for semantic segmentation algorithms, including the following modules:

[0070] The AXI bus unit is used to realize bidirectional data transmission between the PS and PL ends. It receives the network model weights, feature data and network layer parameter information sent by the PS end, and sends the final inference results of the PL end back to the PS end.

[0071] The control unit is used to read the network layer information being calculated, decode it, and generate control and configuration information for each module to realize the reading and writing of feature maps and the reading of weight data, ensuring the smooth execution of data reception, calculation, and transmission tasks.

[0072] The caching unit includes a feature map caching module and a weight caching module, which are used to cache externally written feature map and weight data as well as intermediate features output by the parallel computing unit.

[0073] The preprocessing unit generates the access addresses for the feature map and weight storage modules, and detects filled regions in real time, providing the computation unit with the feature map windows required for different convolution calculation types. (See reference...) Figure 4 The preprocessing unit includes a row buffer module and an optimized register cache matrix. It uses the row and column indices of the input features to detect in real time whether the current processing position is a filled region. If it is determined to be a filled position, the register cache matrix still caches valid input features at the corresponding position, while simultaneously generating a zero value and outputting it to the pipelined computation unit. If it is a valid pixel region, the cached value of the register matrix is ​​output to the pipelined computation unit. This method avoids caching and retrieving zero-value fillers, reducing cache requirements and improving computational efficiency.

[0074] The pipelined multiplexing computation unit is used to realize layer fusion convolutional pipelined computation between continuous network layers. The first stage uses four 3×3 PE arrays with a depth of 16, and the second stage uses 32 3×3 PE arrays with a depth of 2; refer to Figure 5 When the output features of the first-level computing unit can form a complete feature window, the convolution calculation of the second-level computing unit can be started in parallel, thereby greatly reducing the storage requirements of intermediate features of adjacent layers and improving computational efficiency.

[0075] The residual computation unit uses four 1×1 PE arrays with a depth of 2. By reusing the input data of the first-level computation unit, it realizes the residual connection operation in the backbone of the CNN branch network.

[0076] Bilinear Upsampling Unit: Responsible for implementing bilinear interpolation upsampling operations on the output features of the fourth layer in the model. This computation unit adopts a three-level row buffer module design to realize pipelined computation of the upsampled data stream.

[0077] The embodiments of the present invention have been described in detail with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A real-time semantic segmentation method for intelligent driving adapted to complex climates, characterized in that, Includes the following steps: Step S1: Based on the encoder-decoder architecture, construct the network structure of a single-branch CNN-type semantic segmentation network; Step S2: Build a Transformer-type teacher branch for the encoder-decoder architecture. Train it to convergence on a complex climate dataset consisting of the normal climate dataset Cityscapes and the adverse climate dataset ACDC to obtain a converged Transformer teacher branch. Step S3: Construct a boundary enhancement label generation module and a boundary detail loss function to alleviate the training imbalance problem between normal and adverse weather conditions; Step S4: Based on the network structure of a single-branch CNN-type semantic segmentation network, a generation denoising-weighted distillation strategy is constructed. Gaussian noise is added to the intermediate features of the CNN student branch for denoising. Then, channel weighting is achieved through the channel attention mechanism. Combined with KL divergence loss, the CNN student branch learns the remote context features of the Transformer teacher branch. Step S5: The constructed single-branch CNN-type semantic segmentation network is used as the student branch. Under the guidance of the Transformer teacher branch that converged in step S2, the boundary detail loss and the generation denoising-weighted distillation strategy are fused for joint training. The intelligent driving scene image under complex weather conditions is input into the trained and converged single-branch CNN-type semantic segmentation network, and the pixel-level semantic segmentation result is output to complete the real-time semantic segmentation.

2. The real-time semantic segmentation method for intelligent driving adapted to complex climates according to claim 1, characterized in that, In step S1, the encoder of the single-branch CNN-type semantic segmentation network adopts a ResNet-like residual backbone network. The intermediate features of the encoder are divided into four layers, labeled as layer 1 to layer 4. After adjusting the number of channels of the features of layer 4 through 1×1 convolution, bilinear upsampling is performed to the size of the features of layer 2, and channel splicing and fusion are performed. The fused features are sent to the decoder to output the segmentation result.

3. The real-time semantic segmentation method for intelligent driving adapted to complex climates according to claim 1, characterized in that, In step S2, the Transformer-type teacher branch adopts an encoder-decoder architecture adapted to a single-branch CNN-type semantic segmentation network. The encoder's backbone outputs four layers of intermediate features, labeled as layers 1 to 4. The features of layers 2 to 4 are bilinearly upsampled to the size of the features of layer 1, and then channel-by-channel splicing and fusion are performed. The fused features are then sent to the decoder to output the segmentation result.

4. The real-time semantic segmentation method for intelligent driving adapted to complex climates according to claim 1, characterized in that, In step S3, the boundary enhancement label generation module and the boundary detail loss function are specifically as follows: For the actual labels used in training, the boundary enhancement label generation module uses two-dimensional boundary convolution kernels, horizontal convolution kernels, and vertical convolution kernels with fixed parameters to acquire multi-scale detail information with different step lengths. After upsampling, dynamic weighted summation is performed through 1×1 convolution, and the result is converted into binary boundary enhancement labels according to the set threshold. The calculation formula is as follows: ； In the formula, This represents the boundary enhancement label in the i-th row and j-th column. This represents the feature in the i-th row and j-th column obtained by calculating and fusing two-dimensional boundary convolution kernels, horizontal convolution kernels, and vertical convolution kernels with fixed parameters; The DICE loss and binary cross-entropy loss are calculated using the boundary enhancement labels and the features from the second layer of the CNN branch, respectively. The boundary detail loss is then obtained from the binary cross-entropy loss and the DICE loss, as shown in the following formula: ； ； In the formula, H and W are the height and width of the second-layer feature and boundary enhancement label, respectively. It is a DICE loss. It is a binary cross-entropy loss. It is a loss of boundary details. The segmentation prediction is obtained from the output features of the second-layer encoder. For the generated boundary enhancement labels, For feature height, This represents the feature width.

5. The real-time semantic segmentation method for intelligent driving adapted to complex climates according to claim 1, characterized in that, In step S4, the construction of the denoising-weighted distillation strategy specifically involves: adding the features of the 3rd and 4th layers of the CNN branch to Gaussian noise with a mean of 0 and a variance of 1, respectively; using a backbone feature alignment module to achieve denoising and channel dimension alignment, so that the number of channels of the features of the 3rd and 4th layers of the CNN branch is consistent with the number of channels of the 3rd and 4th layers of the teacher branch; then, using a channel attention mechanism to perform channel weighting on the features of the 3rd and 4th layers of the CNN branch, and calculating the KL divergence between the weighted features and the teacher features of the 3rd and 4th layers, respectively, as the distillation loss.

6. The real-time semantic segmentation method for intelligent driving adapted to complex climates according to claim 1, characterized in that, The branch CNN semantic segmentation network is the core inference network of the semantic segmentation method; the Transformer teacher branch, the boundary enhancement label generation module, and the backbone feature alignment module in the generation denoising-weighted distillation strategy are all auxiliary modules in the network training stage and do not participate in real-time inference during hardware deployment.

7. A hardware acceleration system for real-time semantic segmentation of intelligent driving adapted to complex climates, characterized in that, The inference process of a single-branch CNN semantic segmentation network that can be deployed after training is implemented based on FPGA in any of the semantic segmentation methods described in claims 1-6. The hardware acceleration system consists of two parts: the PS and PL hardware accelerators. The PS is used to implement software-level control and post-processing, including the pre-storage of input images and weight parameters, the final post-processing of segmentation results, and communication with the PL. The PL terminal implements a fully pipelined hardware circuit design, specifically adapted for the forward inference computation of the aforementioned CNN branch semantic segmentation network, including: The AXI bus unit is used to realize bidirectional data transmission between the PS and PL ends. It receives the network model weights, feature data and network layer parameter information sent by the PS end, and sends back the 4th layer features and the final inference result of the PL end to the PS end. The control unit is used to read the network layer information being calculated, decode it, and generate control and configuration information for each module to realize the reading and writing of feature maps and the reading of weight data, ensuring the smooth execution of data reception, calculation, and transmission tasks. The caching unit includes a feature map caching module and a weight caching module, which are used to cache externally written feature map and weight data as well as intermediate features output by the parallel computing unit. The preprocessing unit is used to generate the access addresses of the feature map and weight storage module, and to detect the padding region in real time. It provides the feature map window required for convolution calculation to the computing unit, while avoiding caching and reading of padding zero values, thus improving computing efficiency. The pipelined computing unit is used to realize the pipelined computation of convolutional layer fusion between continuous network layers. The first stage uses 4 groups of 3×3 PE arrays with a depth of 16, and the second stage uses 32 groups of 3×3 PE arrays with a depth of 2. When the output features of the first-stage computing unit form a complete feature window, the convolutional computation of the second-stage computing unit can be started in parallel. The residual computation unit uses four 1×1 PE arrays with a depth of 2. By reusing the input data of the first-level computation unit, it realizes the residual connection operation in the backbone of the CNN branch network. Bilinear Upsampling Unit: Responsible for implementing bilinear interpolation upsampling operations on the output features of the fourth layer in the model. This computation unit adopts a three-level row buffer module design to realize pipelined computation of the upsampled data stream.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

Images