Automatic driving visual joint perception method and device and medium

Abstract

The application discloses an automatic driving visual joint perception method and device and a medium, for a given traffic scene image input, different level backbone feature representations are obtained; the multi-level backbone features are preliminarily aggregated to obtain coarse multi-scale feature representations; dynamic convolution is adopted to adaptively obtain feature weighting coefficients of each level, and the final encoding representation is obtained by weighting and summing the features of different scale levels according to the obtained weights; the encoding representation is taken as shared feature input of a target detection, drivable area segmentation and lane line detection decoding prediction network; in the target detection decoding prediction network part, classification and positioning subtask gradient flows in target detection are separately encoded in the spatial dimension, the target detection prediction network is decoupled into two task branches of classification and positioning, high-level semantic features strongly related to target categories and high-resolution dense encoding features related to target boundaries are respectively generated according to the shared feature input, and task conflicts between the classification and positioning tasks are reduced.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and more specifically, relates to an autonomous driving vision-based joint perception method, device, and medium. Background Technology

[0002] Autonomous driving technology can effectively assist in comprehensive urban traffic management, enabling safer, smarter, and more connected vehicle management, and providing strong support for the sustainable development of related industries and social harmony and stability. As a key component of autonomous driving systems, visual perception technology can capture information about the vehicle's surrounding environment in traffic scenarios, providing the control system with sufficient and accurate prior information to assist the driving vehicle in making safe decisions, and has significant application value in fields such as intelligent vehicle networking.

[0003] Autonomous driving visual perception technology mainly includes different tasks such as object detection, drivable area segmentation, and lane detection. Thanks to the widespread application of deep learning technology in visual driving perception, algorithms for object detection, drivable area segmentation, and lane detection have all made significant progress, playing a crucial role in subsequent vehicle decision-making. From the perspective of scene understanding, object detection technology mainly provides the relative position information of traffic targets such as pedestrians and vehicles in the driving scene; drivable area segmentation and lane detection technologies mainly provide the specific lane division methods and drivability judgment information in the driving environment, which can usually be modeled as semantic segmentation tasks. These three tasks are independent of each other, but they also have strong feature correlations. While using multiple complex single-task models to infer for each scene task can meet the feature requirements of different tasks, it also consumes a lot of time and computing resources. In real and ever-changing traffic environments, driving vehicles need to make safe and accurate driving decisions quickly. However, the limited computing resources and memory of mobile in-vehicle systems, to some extent, limit the algorithmic complexity of autonomous driving visual perception models. Therefore, autonomous driving visual perception systems need to achieve an effective trade-off between real-time performance and accuracy.

[0004] Against this backdrop, multi-task learning offers a possible effective trade-off between performance and efficiency. Compared to single-task algorithms, multi-task learning algorithms can leverage the complementarity between multiple tasks, share encoding features across network branches, and simultaneously complete two or more tasks, thereby assisting vehicles in real-time decision-making with limited computing resources. The paper "Teichmann M, Weber M, Zoellner M, et al. MultiNet: Real-time joint semantic reasoning for autonomous driving[C]. 2018 IEEE Intelligent Vehicles Symposium, Changshu, Suzhou, China: IEEE, 2018: 1013-1020." applies an end-to-end multi-task encoding / decoding structure to a joint driving perception system consisting of image classification, object detection, and semantic segmentation. With a small number of parameters introduced, it surpasses the performance of most single-task algorithms, but loses considerable detail while acquiring contextual information. To further enhance the ability to represent details, the paper "Qian Y, Dolan JM, Yang M. DLT-Net: Joint detection of drivable areas, lanelines, and traffic objects[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 21(11): 4670-4679." proposes a joint perception algorithm for drivable area segmentation, lane detection, and target detection based on multi-task learning. It uses a Feature Pyramid Network (FPN) to fuse high-level abstract semantics and texture details at different scales, sharing parameters at the decoding end through interactive information flow. This simultaneously achieves target detection, drivable area segmentation, and lane detection in autonomous driving scenarios, but its real-time application is not yet mature. Building on this, the paper "WuD, Liao MW, Zhang WT, et al. YOLOP: You only look once for panoptic driving perception[J]. Machine Intelligence Research, 2022: 550-562." proposes the YOLOP algorithm, which simplifies the encoding and decoding structure and achieves better performance in real-time aspects compared to mainstream algorithms. However, existing multi-task visual joint perception algorithms still suffer from feature requirement conflicts within each task during the decoding and prediction process.In object detection tasks, existing algorithms often employ coupled decoding networks, failing to consider the different spatial granularity feature prediction requirements and internal competition between classification and localization subtasks within the object detection task. In drivable area segmentation and lane detection tasks, existing algorithms neglect the semantic feature differences between semantically consistent main regions and semantically abrupt edge pixels. Therefore, prediction conflicts between classification confidence and regression scores, as well as coarse edge segmentation results, persist during detection and segmentation. Summary of the Invention

[0005] This invention addresses the aforementioned problems in the prior art. Therefore, there is a need for an autonomous driving vision-based joint perception method, device, and medium to simultaneously achieve target detection, drivable area segmentation, and lane detection. To address the differences and conflicts in classification and localization task requirements in target detection, a classification and localization branch is established based on the task decoupling concept. Based on shared feature inputs, different feature encoding methods are used to specifically design the prediction inputs for the two types of tasks, alleviating feature granularity conflicts between tasks. Simultaneously, an auxiliary Intersection over Union (IoU) perception prediction branch is added to the localization part to strengthen the correlation between the two types of tasks and increase the contribution ratio of the localization sub-task, thereby improving localization accuracy. To address the coarse pixel mask segmentation problem in drivable area segmentation and lane detection tasks, a feature decoupling concept is used for spatial feature separation and reconstruction. The low-frequency main part and high-frequency edge part of the predicted target are decomposed and modeled. An edge loss function guides the local training and learning of the foreground target, thereby improving overall prediction performance.

[0006] According to a first aspect of the present invention, an autonomous driving vision-based joint perception method is provided, the method comprising:

[0007] Based on traffic road images, different levels of backbone features are obtained. These different levels of backbone features are aggregated to obtain multi-scale features. Based on the weighting coefficients of each level of features obtained by dynamic convolutional adaptation, the multi-scale features are weighted and summed to obtain multi-scale fusion coding features.

[0008] An object detection decoding network is constructed, which includes a classification branch and a localization branch. The object detection decoding network takes the multi-scale fusion coding features as input and generates context coding features and multi-level spatial texture detail coding features based on the multi-scale fusion coding features. The context coding features and multi-level spatial texture detail coding features are respectively input to the classification branch and the localization branch for prediction to obtain classification features and localization features. An IoU perception prediction branch is used for auxiliary learning.

[0009] A dual-branch semantic segmentation decoding network is constructed. The dual-branch semantic segmentation decoding network uses the low-resolution features and high-resolution features in the multi-scale fusion coding features as inputs and feeds them into the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are obtained through the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are fused to obtain the target features. The dual-branch semantic segmentation decoding network is trained and learned using the edge loss function and the overall segmentation loss function.

[0010] Furthermore, the context-encoded features are generated using the following method:

[0011] Obtaining multi-scale fusion coding features F l F l+1 , l represents the feature level at which the fused feature is located, where F l For a fusion feature that is 1 / 16 the resolution of the original image, F l+1 The fusion feature is 1 / 32 of the original image resolution;

[0012] For F l+1 The downsampling feature F is obtained by performing downsampling. l+2 , where F l+2 The fusion feature is 1 / 64 the resolution of the original image;

[0013] F l F l+1 F l+2 As multi-scale feature input, semantic context feature encoding at the l+n pyramid level is achieved, resulting in context-encoded feature F'. l+n :

[0014]

[0015] Where n = [0,1,2], CBS represents the processing procedure performed by a convolutional normalization activation function combination module consisting of a 1×1 convolutional layer with a stride of 1, a batch normalization layer, and an activation function layer, f down This indicates a 1×1 convolutional downsampling layer with a stride of 2.

[0016] Furthermore, the classification features are obtained through the following method:

[0017] The classification branch f consists of two convolutional layers. cls (·) Predict the classification score, and after feature recombination, obtain the classification feature with size B×H×W×C, where B is the batch number, H and W are the height and width of the feature map respectively, and C is the number of categories.

[0018] Furthermore, the multi-level spatial texture detail encoding features are generated using the following method:

[0019] Obtaining multi-scale fusion coding features F l-1 F l F l+1 , l represents the feature level at which the fused feature is located, where F l-1 For a fusion feature that is 1 / 8 the resolution of the original image, F l For a fusion feature that is 1 / 16 the resolution of the original image, F l+1 The fusion feature is 1 / 32 of the original image resolution;

[0020] The multi-level spatial texture detail encoding feature F' is calculated using the following formula. l+m :

[0021]

[0022] Where m = [-1, 0, 1], Concat represents the feature concatenation operation, CBS represents the processing procedure performed by a convolutional normalization activation function combination module consisting of a 1×1 convolutional layer with a stride of 1, a batch normalization layer, and an activation function layer, and f down f represents a 1×1 convolutional downsampling layer with a stride of 2. up This is a bilinear interpolation upsampling operator with an upsampling factor of 2.

[0023] Furthermore, the localization branch includes a regression branch f. reg (·), Target confidence branch f obj (·) and IoU perception prediction branch f IoU (·), the localization features are obtained through the following method:

[0024] The multi-level spatial texture detail encoding features are input into the localization branch to predict the target bounding box. After feature recombination, coordinate predictions of size B×H×W×4, confidence predictions of size B×H×W×1, and intersection-union ratio (IoU) score predictions of size B×H×W×1 are obtained, where B is the batch number, and H and W are the height and width dimensions of the feature map, respectively.

[0025] Furthermore, the auxiliary learning using the IoU-aware prediction branch includes:

[0026] During the training phase, the IoU-aware prediction branch uses the Binary Cross-Entropy (BCE) function to calculate the loss L for predicting positive samples. IoU :

[0027]

[0028]

[0029] Where N is the number of positive samples, s is any number of positive samples, pos is the set of positive samples, BCE(·) is the binary cross-entropy function, and for each positive sample, the predicted IoU score obtained by the IoU-aware prediction branch is expressed as: For the corresponding prediction box With real frame The true value of the IoU score is calculated using the overlap(·) function;

[0030] During the model inference phase, the original classification score, target confidence score, and IoU perceptual prediction score are multiplied to obtain the final confidence score S. fin :

[0031]

[0032] For each positive sample s, the corresponding classification score, target confidence score, and IoU perception prediction branch prediction score are cls, respectively. s obj s IoU s The hyperparameter λ takes values ​​in the range (0,1).

[0033] Furthermore, the low-resolution and high-resolution features in the multi-scale fusion coding features are used as inputs and fed into the low-resolution and high-resolution branches respectively. Low-frequency main features and high-frequency edge features are obtained through the low-resolution and high-resolution branches respectively. The low-frequency main features and high-frequency edge features are then fused to obtain the target features, including:

[0034] Convolution and bilinear interpolation upsampling are performed on low-resolution and high-resolution features respectively to obtain intermediate feature maps. The low-resolution and high-resolution features are then concatenated with the intermediate feature maps to obtain an image frame sequence. The image frame sequence is processed using an optical flow field migration operator to obtain the motion offset of each pixel. An optical flow strain feature map is determined based on the motion offset. The pixel value of each pixel in the optical flow strain feature map is half of the sum of the original pixel value and the pixel value corresponding to its motion offset.

[0035] The optical flow strain feature map is stitched together with low-resolution and high-resolution features respectively. Differentiable bilinear interpolation is performed on multiple neighboring pixels around the feature stitching result by optical flow field distortion operator, and the corresponding pixel values ​​of multiple adjacent pixels are weighted and summed to obtain low-frequency main features. The low-frequency main features include low-resolution low-frequency main features and high-resolution low-frequency main features.

[0036] The high-frequency edge features include low-resolution high-frequency edge features and high-resolution high-frequency edge features. The low-resolution features and high-resolution features are subtracted pixel by pixel from the corresponding low-resolution low-frequency main features and high-resolution low-frequency main features to obtain the low-resolution high-frequency edge features and high-resolution high-frequency edge features.

[0037] In the direction from high-resolution branch to low-resolution branch, two convolution operators are used to downsample the high-resolution high-frequency edge features. After adding them with the low-resolution high-frequency edge features, a bilinear interpolation upsampling operation is used to upsample the sum, resulting in the downsampled matching feature output in the corresponding direction. In the direction from low-resolution branch to high-resolution branch, a bilinear interpolation upsampling operation is used to upsample the low-resolution high-frequency edge features. The feature weight map is obtained through an activation function and multiplied with the high-resolution high-frequency edge features in a weighted manner to obtain the upsampled matching feature output in the corresponding direction. The downsampled matching feature output and the upsampled matching feature output are fused using a feature concatenation method. After continuous upsampling operations, the final high-frequency edge features are obtained.

[0038] In the direction from high-resolution branch to low-resolution branch, two convolution operators are used to downsample the high-resolution low-frequency main features. After adding them with the low-resolution low-frequency main features, a bilinear interpolation upsampling operation is used to upsample the sum, resulting in the downsampled matching feature output in the corresponding direction. In the direction from low-resolution branch to high-resolution branch, a bilinear interpolation upsampling operation is used to upsample the low-resolution low-frequency main features. The feature weight map is obtained through an activation function and multiplied with the high-resolution low-frequency main features in a weighted manner to obtain the upsampled matching feature output in the corresponding direction. The downsampled matching feature output and the upsampled matching feature output are fused using a feature concatenation method. After continuous upsampling operations, the final low-frequency main features are obtained.

[0039] The final high-frequency edge features and the final low-frequency main features are added pixel by pixel to obtain the target features.

[0040] Furthermore, the dual-branch semantic segmentation decoding network is trained using edge loss functions and overall segmentation loss functions, including:

[0041] The loss function is expressed as:

[0042]

[0043] Among them, L total L full L edge These are the total loss function, the overall segmentation loss function, and the marginal loss function, respectively. The overall segmentation loss function uses the binary cross-entropy function, F'. full Indicate target features, For the corresponding complete truth labeling diagram, F' edge Indicates the final high-frequency edge features, The corresponding edge truth label map is calculated using the following formula:

[0044]

[0045] Where p is any pixel value in the edge label matrix. For the complete truth label graph edge nearest neighbor set, δ is the Euclidean distance between the pixel value and the set of nearest neighbors on the edge of the complete ground truth label image. δ is a set threshold. When the Euclidean distance is not greater than δ, the value of the corresponding pixel p is set to 1. When the Euclidean distance is less than δ, the value of the corresponding pixel p is set to 0.

[0046] Edge loss function L edge The calculation formula is as follows:

[0047]

[0048] Where I represents the image input, p i The edge truth label representing pixel i, q i This represents the edge prediction result for pixel i.

[0049] According to a second technical solution of the present invention, an autonomous driving vision-based joint perception device is provided, the device comprising:

[0050] The fusion feature calculation module is configured to obtain backbone features at different levels based on traffic road images, aggregate the backbone features at different levels to obtain multi-scale features, and perform weighted summation on the multi-scale features based on the weighting coefficients of each level of features obtained by dynamic convolutional adaptation to obtain multi-scale fusion coding features.

[0051] The target category location recognition module is configured to construct a target detection decoding network, which includes a classification branch and a localization branch. The target detection decoding network takes the multi-scale fusion coding features as input and generates context coding features and multi-level spatial texture detail coding features based on the multi-scale fusion coding features. The context coding features and multi-level spatial texture detail coding features are respectively input to the classification branch and the localization branch for prediction to obtain classification features and localization features. An IoU perception prediction branch is used for auxiliary learning.

[0052] The drivable area and lane line recognition module is configured to construct a dual-branch semantic segmentation and decoding network. The dual-branch semantic segmentation and decoding network uses the low-resolution and high-resolution features in the multi-scale fusion coding features as inputs and feeds them into the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are obtained through the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are fused to obtain the target features. The edge loss function and the overall segmentation loss function are used to guide the dual-branch semantic segmentation and decoding network to train and learn.

[0053] According to a third technical solution of the present invention, a readable storage medium is provided, wherein the readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described above.

[0054] The present invention has at least the following beneficial effects:

[0055] 1. This invention adopts the concept of task decoupling and makes targeted designs for the vision joint perception decoding prediction network for autonomous driving;

[0056] 2. In object detection tasks, the sub-tasks of classification and localization have different requirements for feature granularity. Classification tasks require more contextual information to determine the object category, while bounding box regression aims to obtain more edge texture information for accurate target boundary localization. However, existing algorithms often use coupled detection heads, achieving joint optimization of object category and bounding box prediction by sharing convolutional layer parameters. This approach fails to consider the spatial misalignment of feature regions required by different tasks, thus limiting detector performance improvement to some extent. Therefore, this invention comprehensively considers the different feature granularity requirements of classification and localization, constructing a hierarchical semantic enhancement module and a spatial information refinement module for separate feature encoding. This generates high-level semantic features strongly correlated with the target category and high-resolution densely encoded features related to the target boundary, respectively. By aggregating feature maps at different semantic levels, task conflicts between classification and localization are reduced, mitigating over-prediction of foreground regions caused by the fading of significant semantic cues and incomplete target structure caused by feature extraction downsampling. Based on this, in order to further strengthen the correlation between the two decoupled branches of classification and localization, an IoU perception prediction module is designed in the localization part to enhance the correlation between classification confidence and bounding box regression score, and increase the contribution ratio of bounding box regression subtask, thereby improving localization accuracy.

[0057] 3. In high-resolution autonomous driving traffic scenarios, the relationship between background structure information and driving vehicle targets is complex and variable. Furthermore, influenced by different shooting conditions and climate changes, lane edges are prone to shadows and vehicle occlusion, posing significant challenges to drivable area segmentation and lane line detection. From the perspective of foreground target spatial structure, lane lines and drivable areas can be decomposed into internal main regions and boundary regions. Due to interference from surrounding background information, lane boundary regions exhibit semantic jumps, while internal main pixels maintain similar semantic motion trends, resulting in significant semantic differences. Existing algorithms do not consider these semantic differences, leading to inaccurate boundary region segmentation results. Pixel mask boundary regions sometimes exhibit "irregular patches" and inaccurate segmentation edges. To address the coarse segmentation problem of predicted pixel masks, this invention constructs a dual-resolution decoupled branch network. Based on the optical flow field migration concept, it decomposes and models the foreground target features in traffic scene images, dividing the target features into low-frequency main regions and high-frequency boundary regions according to motion semantic information. Based on the overall segmentation loss function, edge loss is introduced to guide the foreground target to achieve training and learning from local to global, realize the gradual optimization of the predicted target body and edge parts, improve the problem of irregular patches and inaccurate edge segmentation inside the segmentation pixel mask, and improve the overall detection and segmentation performance.

[0058] 4. The present invention decouples task features according to the different feature requirements of each perception task, which has a certain degree of generalization and can be applied to most autonomous driving vision joint perception models based on encoding and decoding structures.

[0059] 5. When applied to real driving scenarios, this invention can effectively capture information about the surrounding environment of the vehicle in the road scene, effectively balance the accuracy and real-time performance of the visual perception system, achieve high-quality and efficient visual joint perception, effectively save computing resources and memory usage of the mobile driving system, and facilitate practical deployment and application. Attached Figure Description

[0060] Figure 1 An overall flowchart of an autonomous driving vision-based joint perception method according to an embodiment of the present invention is shown;

[0061] Figure 2 A structural diagram illustrating the implementation of an autonomous driving vision-based joint perception method according to an embodiment of the present invention is shown.

[0062] Figure 3 A feature encoding structure diagram (n=2) of the hierarchical semantic enhancement module (HSEM) according to an embodiment of the present invention is shown;

[0063] Figure 4 A feature encoding structure diagram (m=0) of the Spatial Information Refinement Module (SIRM) according to an embodiment of the present invention is shown;

[0064] Figure 5 The overall structure diagram of a dual-branch semantic segmentation and decoding network based on task feature decoupling according to an embodiment of the present invention is shown.

[0065] Figure 6 A structural diagram of an autonomous driving vision-based joint perception device according to an embodiment of the present invention is shown;

[0066] Figure 7 The multi-task vision-driven joint perception method YOLOP according to an embodiment of the present invention is shown, along with visualization results of the method in different scenarios. Detailed Implementation

[0067] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples, but this is not intended to limit the present invention. If there is no necessary sequential relationship between the various steps described herein, the order in which they are described as examples should not be considered a limitation. Those skilled in the art should understand that the order can be adjusted, as long as it does not disrupt the logical consistency between them and render the entire process impossible.

[0068] Figure 1 A flowchart illustrating an autonomous driving vision-based joint perception method according to an embodiment of the present invention is shown, as follows: Figure 1 As shown, this embodiment of the invention provides an autonomous driving vision-based joint perception method, which includes the following steps S100-S300.

[0069] Step S100: Obtain backbone features at different levels based on traffic road images, aggregate the backbone features at different levels to obtain multi-scale features, and perform weighted summation on the multi-scale features based on the weighting coefficients of each level of features obtained by dynamic convolutional adaptation to obtain multi-scale fusion coding features.

[0070] For example, the traffic road image described in this embodiment can be acquired by a vehicle-mounted camera module. The purpose of this step is to perform preliminary processing on the traffic road image to obtain multi-scale fusion coding features. These multi-scale fusion coding features can be represented in graph form and used as input in the subsequent data processing steps S200 and S300.

[0071] In this embodiment, multi-scale fusion coding features can be acquired through an encoder network. Specifically, for a given traffic road image input in an autonomous driving scenario, a lightweight backbone network CSPDarkNet-53 is first used to obtain backbone feature representations at different levels. The multi-level backbone features are then input into a feature pyramid pooling module and a feature pyramid module for preliminary aggregation, resulting in a coarse multi-scale feature representation. Dynamic convolution adaptively obtains the weighting coefficients for each level of features, and the features at each level are weighted and summed to obtain the updated multi-scale fusion coding features {F}. n}, including F l-1 F l F l+1 The corresponding resolution sizes of the three feature levels are 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively, where l represents the feature level of the fused feature. Then, the multi-scale fused encoding feature {F} will be used. n} serves as the shared encoding input for the target detection decoding network, the drivable area segmentation decoding network, and the lane line detection decoding network.

[0072] Step S200: Construct an object detection decoding network, which includes a classification branch and a localization branch. The object detection decoding network takes the multi-scale fusion coding features as input and generates context coding features and multi-level spatial texture detail coding features based on the multi-scale fusion coding features. The context coding features and multi-level spatial texture detail coding features are respectively input to the classification branch and the localization branch for prediction to obtain classification features and localization features. An IoU perceptual prediction branch is used for auxiliary learning.

[0073] It should be noted that in practical applications, the target detection and decoding network obtained by auxiliary learning training based on the IoU perception prediction branch can be used to identify targets. The targets mentioned in this paper refer to the targets that autonomous driving needs to identify in traffic road images, including but not limited to pedestrians, vehicles, and traffic structures (such as guardrails and green plants in the middle).

[0074] In this embodiment, in the target detection decoding network, the gradient flows of the classification and localization subtasks in target detection are separated in the spatial dimension, and the original coupled detection head is decomposed into a classification branch f. cls (·) and the localization branch. The localization branch consists of the regression branch f. reg (·), Target confidence branch f obj (·) and IoU perception prediction branch f IoUThe algorithm consists of three parts. To address the different feature requirements of classification and localization tasks, a separate feature encoding approach is adopted. A hierarchical semantic enhancement module and a spatial information refinement module are designed. Based on the original shared features, high-level semantic features strongly correlated with the target category and high-resolution densely encoded features related to the target boundary are generated respectively. These are then fed into the classification and localization sub-task branches to satisfy the feature prediction needs of each branch as much as possible, alleviate the internal competition between sub-tasks, and improve the overall target prediction performance. To strengthen the correlation between the two branches while decoupling detection, an IoU-aware prediction branch is used for auxiliary learning, thereby improving detection accuracy.

[0075] In some embodiments, a hierarchical semantic enhancement module (HSEM) is designed to perform classification feature encoding on shared input features:

[0076] Considering the uneven distribution of salient regions of interest to the classification branches, and the potential for feature redundancy when using complex networks to compute contextual semantic information, this invention designs a hierarchical semantic enhancement module (HSEM) to encode classification features from the original shared input features. Figure 3 The diagram illustrates the specific feature encoding method of the hierarchical semantic enhancement module HSEM when n=2. In the diagram, H represents the HSEM module designed in this invention.

[0077] Considering both computational complexity and performance, dynamic convolution is used to obtain the fused features F at 1 / 16 and 1 / 32 resolution of the original image, respectively. l F l+1 , l represents the feature level at which the fused feature is located, F l For a fusion feature that is 1 / 16 the resolution of the original image, F l+1 This is a fusion feature at 1 / 32 resolution of the original image. Simultaneously, it utilizes the existing Bottleneck CSP module from CSPDarkNet-53, concatenated with a 1×1 convolution operator with a stride of 2, to perform fusion on F... l+1 Performing a 2x downsampling yields a feature F at 1 / 64 the resolution of the original image. l+2 F l F l+1 F l+2 As the multi-scale feature input of the hierarchical semantic enhancement module, it realizes the semantic context feature encoding of the l+n pyramid level, and obtains the multi-level pyramid semantic context encoding feature F'. l+n Where n = [0, 1, 2], specifically:

[0078]

[0079] The Convolutional BatchNorm SiLU (CBS) module consists of a 1×1 convolutional layer with a stride of 1, a batch normalized (BN) layer, and an activation function SiLU layer.down This indicates a 1×1 convolutional downsampling layer with a stride of 2.

[0080] Depend on Figure 3 Thus, the corresponding high-level semantic feature F' at the pyramid level can be obtained. l+2 By downsampling 4 times F l and downsampled by 2 times F l+1 The feature map is calculated by stacking operators such as convolution and pixel-by-pixel addition, so F can be predicted at each pixel position. l 16 adjacent pixels and F l+1 The classification scores of four adjacent pixels are used to reduce the amount of computation and effectively utilize the sparse distribution of the saliency discrimination features of the target object, which helps to obtain better classification results.

[0081] From the overall framework Figure 2 From the target detection decoding network part, we can obtain the multi-level pyramid semantic context encoding features F' generated by the layer-by-layer semantic enhancement module HSEM. l+n Then, these values ​​need to be input into the classification and localization branches respectively. In the classification branch, an f-type convolutional layer consisting of two convolutional layers is used. cls (·) Predict the classification score, and after feature recombination, obtain the classification feature with size B×H×W×C, where B is the batch number, H and W are the height and width of the feature map respectively, and C is the number of categories.

[0082] In some embodiments, in the target detection decoding prediction network part, a spatial information refinement module (SIRM) is designed to encode the shared input features for localization features.

[0083] Compared to the classification branch, the localization branch focuses more on the contour edges and spatial texture details of the target object in the feature map. Therefore, using high-resolution low-level feature maps with more detailed information is helpful for the localization subtask in object detection. However, using all high-resolution features can lead to a waste of additional computing resources and increase model inference time, which is not conducive to the real-time application of the network model. On the other hand, considering that low-resolution high-level semantic features can provide more contextual information, they play an overall guiding role in the detailed information during model training and inference, which can effectively reduce the problem of over-prediction of foreground targets or incomplete target structure. Based on the above analysis, this invention designs a spatial information refinement module SIRM, which uses a combination of high and low resolution features to encode the original shared features for localization, improving the overall localization performance without increasing the amount of computation. For the pyramid level l+m, where m=[-1,0,1], the feature input of SIRM is the fused feature F obtained in step S100. l-1 F l F l+1The three resolutions are 1 / 8, 1 / 16, and 1 / 32 of the original image resolution, respectively, yielding multi-level spatial texture detail encoding features F'. l+m The specific encoding method is as follows:

[0084]

[0085] The Convolutional BatchNorm SiLU (CBS) module consists of a 1×1 convolutional layer with a stride of 1, a batch normalized (BN) layer, and an activation function SiLU layer. down f represents a 1×1 convolutional downsampling layer with a stride of 2. up For a bilinear interpolation upsampling operator with an upsampling factor of 2, for input features F l-1 F l F l+1 The multi-level spatial texture detail encoding features obtained through the SIRM module are F' l+m , m = [-1, 0, 1]. F' l-1 By F l-1 With F l The resolution was calculated using CBS, upsampling, and other operations, and is 1 / 8 of the original image. l '、F' l+1 Each is encoded by the feature F' of the previous level. l-1 F l The recursive calculations yielded resolutions of 1 / 16 and 1 / 32 of the original image, respectively.

[0086] Figure 4 The diagram illustrates the specific encoding method of the SIRM module when m=0. In the figure, S represents the SIRM encoding module, and C represents feature concatenation (Concat). Since each pyramid level is closely related to features at adjacent scales, when m=0, F'... l+m Mainly composed of the encoding result F' of the previous level (m=-1) l-1 and F l+1 It is calculated through stacking operators such as convolution and feature concatenation. This encoding method can effectively preserve the underlying features F without increasing the computational cost too much. l-1 Detailed information within, while also utilizing high-level features F l+1 The semantic context-guided texture detail feature encoding provides more accurate edge contour information for localization tasks. This is achieved by obtaining multi-level spatial texture detail encoding features F' generated by the Spatial Information Refinement Module (SIRM). l+m Then, it is fed into the localization branch to predict the target bounding box, where the localization branch consists of the regression branch f. reg (·), Target confidence branch f obj (·) and IoU perception prediction branch f IoUThe system consists of three parts, each composed of two convolutional layers. After feature recombination, it obtains coordinate predictions of size B×H×W×4, confidence predictions of size B×H×W×1, and intersection-union ratio (IoU) score predictions of size B×H×W×1, where B is the batch size, and H and W are the height and width dimensions of the feature map, respectively.

[0087] In some embodiments, in the object detection decoding prediction network part, an IoU perception prediction branch is designed to enhance the association between the classification and localization subtasks;

[0088] In object detection tasks, the classification subtask and the localization subtask are internally competitive. The bounding box with the best IoU does not necessarily have the highest classification score, and the bounding box with the highest classification score does not necessarily have accurate localization prediction. That is, the detection network suffers from a problem of low correlation between classification confidence and localization accuracy.

[0089] Previous algorithms used CIoU (Complete Intersection over Union) to comprehensively consider the center distance and scale ratio between the ground truth bounding box and the predicted bounding box, calculating the bounding box loss function using the CIoU overlap score. During non-maximum suppression and metric calculation in the model inference phase, the obtained CIoU score was multiplied by the classification score as a metric to rank the predicted bounding boxes. In this process, the CIoU score might be less than 0. To avoid this, previous algorithms manually set the minimum CIoU value to 0, which to some extent disrupted the correlation between classification and localization scores, limiting detector performance improvement.

[0090] On the other hand, considering that the average accuracy calculation in object detection is more closely related to the localization subtask, this invention adds an IoU-aware prediction branch to the localization part of the decoding network to predict the IoU of each prediction box. During the training phase, the IoU-aware prediction branch uses a binary cross-entropy function to calculate the loss L for predicting positive samples. IoU , specifically:

[0091]

[0092]

[0093] Where N is the number of positive samples, s is any number of positive samples, pos is the set of positive samples, and BCE(·) is the binary cross-entropy function. For each positive sample, the predicted IoU score obtained by the IoU-aware prediction branch can be expressed as: For the corresponding prediction box With real frame The ground truth IoU score is calculated using the overlap(·) function. Optimization through objective function training can reduce the gap between the IoU-aware predicted score and the ground truth score, thereby improving positioning accuracy.

[0094] During the model inference phase, the original classification score, target confidence score, and IoU perceptual prediction score are multiplied to obtain the final confidence score S. fin This serves as a new metric for ranking prediction boxes. Specifically:

[0095]

[0096] For each positive sample s, the corresponding classification score, target confidence score, and IoU-aware prediction branch prediction score are cls, respectively. s obj s IoU s The hyperparameter λ ranges from (0,1), and its contribution weights to the final confidence score prediction can be adjusted by controlling the parameter. Using the IoU-aware prediction branch to predict the final confidence score allows for the simultaneous inclusion of the predicted bounding box classification score, the overlap between the predicted and ground truth bounding boxes, the center distance, the aspect ratio, and the predicted IoU in the evaluation metrics. This strengthens the correlation between classification and localization tasks, alleviating the problem of low correlation between classification confidence and localization accuracy. Simultaneously, the IoU-aware prediction branch in the localization part strengthens the localization constraints for the overall prediction, thereby increasing the contribution of the localization task to the overall prediction. Without affecting inference speed, this assists the detection network in improving localization accuracy.

[0097] Step S300: Construct a dual-branch semantic segmentation decoding network. The dual-branch semantic segmentation decoding network uses the low-resolution features and high-resolution features in the multi-scale fusion coding features as inputs and feeds them into the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are obtained through the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are fused to obtain the target features. The edge loss function and the overall segmentation loss function are used to guide the dual-branch semantic segmentation decoding network to train and learn.

[0098] It should be noted that this step is used to achieve drivable area segmentation and lane line detection. In specific applications, the dual-branch semantic segmentation decoding network trained by the above method steps can be used for drivable area segmentation and lane line detection.

[0099] In this embodiment, a dual-branch semantic segmentation and decoding network based on task feature decoupling is designed for the drivable area segmentation and lane detection decoding network. Multi-scale fusion encoding features {F} are employed. n Medium and low resolution features F L With high-resolution feature F HAs input to the decoding network, it is fed into the Low Resolution Branch (LRB) and the High Resolution Branch (HRB), respectively, where F L With F H The corresponding resolutions are 1 / 32 and 1 / 8 of the original image, respectively. In the LRB and HRB dual-resolution branches, the optical flow field migration technique is used to obtain low-frequency main features with internal consistency and high-frequency edge features with pixel-level abrupt changes, respectively. The high-resolution and low-resolution main features F obtained from the two branches are then compared. Hbody F Lbody and edge features F Hedge F Ledge The input is fed into the Feature Fusion Module (FFM) for feature fusion to achieve optimization and update, and then the edge loss function L is used. edge and overall segmentation loss L full Guide the model to train and learn.

[0100] In some embodiments, a dual-branch semantic segmentation and decoding network based on task feature decoupling is constructed in the drivable area segmentation and lane line detection decoding prediction network part.

[0101] In drivable region segmentation and lane detection tasks, the internal pixels of the two foreground target segmentation masks (drivable region and lane lines) have semantic consistency, while semantic jumps occur at the boundary between the foreground target and the background, exhibiting significant semantic changes compared to the internal pixels. Considering that optical flow can model the motion state and segmentation structure information of the foreground target, obtaining pixels with the same motion trend, and that the elongated edges of the foreground target need to be combined with both high-level semantic information and low-level texture details to ensure prediction accuracy, this invention constructs a dual-branch semantic segmentation decoding network based on task feature decoupling. It employs a feature decoupling method based on optical flow field distortion prediction to decompose the foreground target features, obtaining low-frequency main pixel regions with semantic consistency and high-frequency edge pixel regions with semantic jumps.

[0102] The overall structure of the dual-branch semantic segmentation and decoding network based on task feature decoupling constructed in this invention is as follows: Figure 5 As shown, it mainly consists of two parts: a dual-resolution decoupling branch and a feature fusion structure. The feature input at the top of the figure is the high-resolution feature F. H The branch is a high-resolution branch, and the feature input below is a low-resolution feature F. L The branch is the low-resolution branch, and the feature fusion module FFM used in the feature fusion structure is as follows: Figure 5 As shown in the dashed box on the right.

[0103] For the dual-branch semantic segmentation and decoding network designed in this invention, firstly, based on the multi-scale fusion coding features {F n}. Select high-resolution features F at 1 / 8 and 1 / 32 resolution. H and low-resolution features F L This is used as the feature input for the high-resolution and low-resolution branches, thereby obtaining low-frequency main features and high-frequency edge features in the foreground target. Then, the high-resolution and low-resolution main features F obtained from different resolution branches are combined... Hbody F Lbody and edge features F Hedge F Ledge The features are gradually optimized and updated by fusing them separately using the FFM module. Finally, the updated low-frequency main features F are... b ' ody and the final high-frequency edge feature F' edge The complete target feature F' is obtained by adding pixels one by one. full The final foreground target probability prediction result is obtained by stacking three consecutive upsampling layers. The final feature result obtained after the upsampling operation maintains the same resolution as the input driving scene image.

[0104] Inspired by optical flow estimation algorithms in video sequence prediction tasks, in the dual-resolution decoupling branch, 3×3 convolutions with a stride of 2 are used to apply the high-resolution features F. H and low-resolution features F L Preliminary processing is performed, and intermediate feature maps are obtained through bilinear interpolation upsampling. The original features and the corresponding upsampled intermediate feature maps are packaged into an image frame sequence and input into the optical flow prediction module Flow to coarsely estimate the inter-frame velocity vector V of the semantic features. Taking the high-resolution branch as an example, the specific calculation method is as follows:

[0105] V = f(Concat(F) H ,f up (f1(F H )))

[0106]

[0107] Among them, F H For high-resolution features, f1 is a 3×3 convolution operator with a stride of 2. up f represents the bilinear interpolation upsampling operation. up (f1(F HThe first part, F, represents the intermediate feature map obtained after the above operations on the high-resolution branch feature input, with the same size as the high-resolution feature input. `Concat` represents the feature concatenation operation, and `f` is the optical flow field migration operator, which is actually a 3×3 standard convolution with a stride of 1. After obtaining the motion migration V(p) of each pixel p in the image, optical flow field migration mapping needs to be performed on each grid pixel p in the original feature map to achieve feature update. The updated feature map F... H The pixel value of each pixel point p' in (p') can be represented as half of the sum of the pixel values ​​of the original pixel p and its corresponding motion offset V(p).

[0108] The extracted optical flow strain feature map was then compared with the high-resolution feature F. H and low-resolution features F L Feature stitching operations are performed separately, denoted as C in the figure. The coarse result obtained after stitching is refined using the optical flow field warp operator. By controlling the feature pixel vectors to point inwards towards the target object through the optical flow offset field, more attention is paid to consistent semantic motion information, thereby obtaining the densely distributed low-frequency main feature part F. body Taking the high-resolution branch as an example, the specific calculation process is as follows:

[0109] F' H (p”)=Concat(F H (p),F H (p'))

[0110]

[0111] Among them, F H (p) represents the original high-resolution feature input, F H (p') represents the corresponding optical flow strain feature. The two are concatenated to obtain the warp operator feature input F'. H (p”). For F' H In N(p”), each pixel grid p” is processed by the warp operator to perform differentiable bilinear interpolation on its four neighboring pixels, where N(p”) is the set of the corresponding neighboring pixels. s This represents any randomly selected neighboring pixel. Then, the pixel values ​​of the four adjacent pixels are weighted and summed to obtain the low-frequency main feature part F. body (p”), the weighting weight is the bilinear kernel weight w corresponding to the spatial pixel grid. ps The feature results obtained in this way can effectively reflect the consistent motion information of the foreground target, and better model the main internal pixels in the target segmentation mask.

[0112] After obtaining the low-frequency main feature portion, the high-resolution feature F is input to the decoupled branch. HLow-resolution features F L Corresponding low-frequency main feature F Hbody and F Lbody By performing pixel-by-pixel subtraction, the high-frequency edge features F with semantic jump characteristics of the corresponding branch can be obtained. Hedge and F Ledge .

[0113] In the feature fusion structure, the main body and edge features obtained from the high- and low-resolution branches are used as feature inputs to the FFM module. The FFM module can be generally divided into target edge fusion and target main body fusion. In the target edge fusion part, a bilateral aggregation approach is adopted, comprehensively considering high-resolution and high-frequency edge features F Hedge and low-resolution high-frequency edge features F Ledge The included contour texture information and abstract semantic information complement each other. Based on the semantic context information, the spatial detail feature prediction output is guided to obtain the final high-frequency edge feature F'. edge The specific calculation process is as follows:

[0114]

[0115]

[0116]

[0117] In the direction from high-resolution branch to low-resolution branch (H→L), two cascaded 3×3 convolution operators f with a stride of 2 are used. down For high-resolution high-frequency edge features F Hedge Perform 4x downsampling and compare with low-resolution high-frequency edge features F Ledge After addition, a bilinear interpolation upsampling operation f is performed. up The fusion result is upsampled by 4 times to obtain the downsampled matching feature output in that direction. In the low-resolution to high-resolution branch direction (L→H), a bilinear interpolation upsampling operation f is used. up For low-resolution high-frequency edge features F Ledge After a 4x upsampling, a feature weight map is obtained by applying the softmax activation function δ, and then compared with the high-resolution high-frequency edge feature F. Hedge Weighted multiplication yields the upsampled matching feature output in that direction. After obtaining the downsampled matching feature output and the upsampled matching feature output respectively and Then, the two are fused using a feature concatenation method, and after three consecutive upsampling steps f 3×up After the operation, the final high-frequency edge feature F' is obtained. edge .

[0118] The target subject fusion part adopts the same structural design, comprehensively considering the contour texture information and abstract semantic information contained in the low-level subject features and high-level subject features, to obtain the final low-frequency subject feature F'. body The calculation method is the same as that for the target edge fusion part. Finally, the updated low-frequency main feature F' is used. body and the final high-frequency edge feature F' edge By adding pixels one by one, the complete target feature F' is obtained. full .

[0119] In some embodiments, an edge loss function is introduced in the drivable area segmentation and lane detection loss function to guide the foreground target to achieve training and learning from local to global.

[0120] In the loss function design section, this invention uses the overall segmentation loss function L... full Add edge loss function L to the basis edge Considering the overall region segmentation mask, edge integration is used to guide the foreground target to undergo local training and learning, alleviating the imbalance and inadequacy in the segmentation of mask edges and the main body, achieving complementary segmentation of the global region and local edges, thereby improving the overall prediction performance. The loss function consists of two parts: an overall segmentation loss function and an edge loss function, specifically:

[0121]

[0122] Among them, L total L full L edge These are the total loss function, the overall segmentation loss function, and the marginal loss function, respectively. The overall segmentation loss function uses the classic binary cross-entropy function. F' full Represents complete target features. This is the corresponding complete truth label diagram. F' edge Indicates the final high-frequency edge features, For the corresponding edge truth label map, by The transformation yields, specifically:

[0123]

[0124] First, create an edge truth label map. Initialize as an all-zero matrix, with size equal to Similarly, p represents any pixel value in the edge label matrix. The `distance_transform_edt` built-in function from the SciPy library is used to calculate the set of nearest neighbors between this pixel and the edge of the complete ground truth label map. European distance When the spacing is not greater than δ, the value of the corresponding pixel p is set to 1; otherwise, it is set to 0, thus realizing the edge ground truth map. Pixel value update; this step sets the threshold δ to 2.

[0125] Edge loss function L edge The calculation formula is as follows:

[0126]

[0127] For image input I, p i The edge truth label representing pixel i, q i This represents the edge prediction result for pixel i. In the drivable area segmentation and lane detection network, the network parameters are continuously optimized by minimizing the edge loss function, thereby improving the overall segmentation performance.

[0128] In summary, this invention designs a decoding prediction network specifically for the feature granularity requirements of different perception tasks, and improves the human-computer interaction capability of the visual joint perception system in real driving scenarios by adopting a task feature decoupling approach, thus providing intelligent vehicles with more realistic and detailed road environment information.

[0129] Furthermore, this invention is applicable to most autonomous driving codec-visual joint perception models, enabling intelligent vehicles to fully understand the surrounding environment information of real road scenarios, thereby making real-time safe driving decisions, improving vehicle driving reliability, and reducing the probability of accidents.

[0130] Figure 6 The diagram shows a structural diagram of an autonomous driving vision-joint perception device according to an embodiment of the present invention. The embodiment of the present invention provides an autonomous driving vision-joint perception device, such as... Figure 6 As shown, the device 600 includes:

[0131] The fusion feature calculation module 601 is configured to obtain different levels of backbone features based on traffic road images, aggregate the different levels of backbone features to obtain multi-scale features, and perform weighted summation on the multi-scale features based on the weighting coefficients of each level of features obtained by dynamic convolutional adaptation to obtain multi-scale fusion coding features.

[0132] The target category location recognition module 602 is configured to construct a target detection decoding network, which includes a classification branch and a localization branch. The target detection decoding network takes the multi-scale fusion coding features as input and generates context coding features and multi-level spatial texture detail coding features based on the multi-scale fusion coding features. The context coding features and multi-level spatial texture detail coding features are respectively input to the classification branch and the localization branch for prediction to obtain classification features and localization features. An IoU perception prediction branch is used for auxiliary learning.

[0133] The drivable area and lane line recognition module 603 is configured to construct a dual-branch semantic segmentation decoding network. The dual-branch semantic segmentation decoding network uses the low-resolution features and high-resolution features in the multi-scale fusion coding features as inputs and feeds them into the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are obtained through the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are fused to obtain the target features. The edge loss function and the overall segmentation loss function are used to guide the dual-branch semantic segmentation decoding network to train and learn.

[0134] In some embodiments, the target category location recognition module is further configured to employ a classification branch f consisting of two convolutional layers. cls (·) Predict the classification score, and after feature recombination, obtain the classification feature with size B×H×W×C, where B is the batch number, H and W are the height and width of the feature map respectively, and C is the number of categories.

[0135] In some embodiments, the target category location recognition module is further configured to generate the multi-level spatial texture detail encoding features by means of the following method:

[0136] Obtaining multi-scale fusion coding features F l-1 F l F l+1 , l represents the feature level at which the fused feature is located, F l-1 For a fusion feature that is 1 / 8 the resolution of the original image, F l For a fusion feature that is 1 / 16 the resolution of the original image, F l+1 The fusion feature is 1 / 32 of the original image resolution;

[0137] The multi-level spatial texture detail encoding feature F' is calculated using the following formula. l+m :

[0138]

[0139] Where Concat represents the feature concatenation operation, and the Convolutional Normalized Activation Function Combination Module (CBS(ConvBatchNorm SiLU)) represents the processing procedure performed by a module consisting of a 1×1 convolutional layer with a stride of 1, a batch normalization layer, and an activation function layer. down f represents a 1×1 convolutional downsampling layer with a stride of 2. up This is a bilinear interpolation upsampling operator with an upsampling factor of 2.

[0140] In some embodiments, the localization branch includes a regression branch f reg (·), Target confidence branch f obj (·) and IoU perception prediction branch f IoU (·), the target category location identification module is further configured to obtain location features through the following method:

[0141] The multi-level spatial texture detail encoding features are input into the localization branch to predict the target bounding box. After feature recombination, coordinate predictions of size B×H×W×4, confidence predictions of size B×H×W×1, and intersection-union ratio (IoU) score predictions of size B×H×W×1 are obtained, where B is the batch number, and H and W are the height and width dimensions of the feature map, respectively.

[0142] In some embodiments, the target category location identification module is further configured to:

[0143] During the training phase, the IoU-aware prediction branch uses the binary cross-entropy function to calculate the loss L for predicting positive samples. IoU :

[0144]

[0145]

[0146] Where N is the number of positive samples, s is any number of positive samples, pos is the set of positive samples, and BCE(·) is the binary cross-entropy function. For each positive sample, the predicted IoU score obtained by the IoU-aware prediction branch is expressed as: For the corresponding prediction box With real frame The true value of the IoU score is calculated using the overlap(·) function;

[0147] During the model inference phase, the original classification score, target confidence score, and IoU perceptual prediction score are multiplied to obtain the final confidence score S. fin :

[0148]

[0149] For each positive sample s, the corresponding classification score, target confidence score, and IoU perception prediction branch prediction score are cls, respectively. s obj s IoU s The hyperparameter λ takes values ​​in the range (0,1).

[0150] In some embodiments, the drivable area and lane line recognition module is further configured to:

[0151] Convolution and bilinear interpolation upsampling are performed on low-resolution and high-resolution features respectively to obtain intermediate feature maps. The low-resolution and high-resolution features are then concatenated with the intermediate feature maps to obtain an image frame sequence. The image frame sequence is processed using an optical flow field migration operator to obtain the motion offset of each pixel. An optical flow strain feature map is determined based on the motion offset. The pixel value of each pixel in the optical flow strain feature map is half of the sum of the original pixel value and the pixel value corresponding to its motion offset.

[0152] The optical flow strain feature map is stitched together with low-resolution and high-resolution features respectively. Differentiable bilinear interpolation is performed on multiple neighboring pixels around the feature stitching result by optical flow field distortion operator, and the corresponding pixel values ​​of multiple adjacent pixels are weighted and summed to obtain low-frequency main features. The low-frequency main features include low-resolution low-frequency main features and high-resolution low-frequency main features.

[0153] The high-frequency edge features include low-resolution high-frequency edge features and high-resolution high-frequency edge features. The low-resolution features and high-resolution features are subtracted pixel by pixel from the corresponding low-resolution low-frequency main features and high-resolution low-frequency main features to obtain the low-resolution high-frequency edge features and high-resolution high-frequency edge features.

[0154] In the direction from high-resolution branch to low-resolution branch, two convolution operators are used to downsample the high-resolution high-frequency edge features. After adding them with the low-resolution high-frequency edge features, a bilinear interpolation upsampling operation is used to upsample the sum, resulting in the downsampled matching feature output in the corresponding direction. In the direction from low-resolution branch to high-resolution branch, a bilinear interpolation upsampling operation is used to upsample the low-resolution high-frequency edge features. The feature weight map is obtained through an activation function and multiplied with the high-resolution high-frequency edge features in a weighted manner to obtain the upsampled matching feature output in the corresponding direction. The downsampled matching feature output and the upsampled matching feature output are fused using a feature concatenation method. After continuous upsampling operations, the final high-frequency edge features are obtained.

[0155] In the direction from high-resolution branch to low-resolution branch, two convolution operators are used to downsample the high-resolution low-frequency main features. After adding them with the low-resolution low-frequency main features, a bilinear interpolation upsampling operation is used to upsample the sum, resulting in the downsampled matching feature output in the corresponding direction. In the direction from low-resolution branch to high-resolution branch, a bilinear interpolation upsampling operation is used to upsample the low-resolution low-frequency main features. The feature weight map is obtained through an activation function and multiplied with the high-resolution low-frequency main features in a weighted manner to obtain the upsampled matching feature output in the corresponding direction. The downsampled matching feature output and the upsampled matching feature output are fused using a feature concatenation method. After continuous upsampling operations, the final low-frequency main features are obtained.

[0156] The final high-frequency edge features and the final low-frequency main features are added pixel by pixel to obtain the target features.

[0157] In some embodiments, the drivable area and lane line recognition module is further configured to:

[0158] The loss function is expressed as:

[0159]

[0160] Among them, L total L full L edge These are the total loss function, the overall segmentation loss function, and the marginal loss function, respectively. The overall segmentation loss function uses the binary cross-entropy function, F'. full Indicate target features, For the corresponding complete truth labeling diagram, F' edge Indicates the final high-frequency edge features, The corresponding edge truth label map is calculated using the following formula:

[0161]

[0162] Where p is any pixel value in the edge label matrix. For the complete truth label graph edge nearest neighbor set, δ is the Euclidean distance between the pixel value and the set of nearest neighbors on the edge of the complete ground truth label image. δ is a set threshold. When the Euclidean distance is not greater than δ, the value of the corresponding pixel p is set to 1. When the Euclidean distance is less than δ, the value of the corresponding pixel p is set to 0.

[0163] Edge loss function L edge The calculation formula is as follows:

[0164]

[0165] Where I represents the image input, pi The edge truth label representing pixel i, q i This represents the edge prediction result for pixel i.

[0166] It should be noted that the device described in this embodiment and the method described earlier belong to the same technical concept and can achieve the same technical effect, which will not be repeated here.

[0167] This invention also provides a readable storage medium storing one or more programs that can be executed by one or more processors to implement the methods described in the above embodiments.

[0168] The following section uses specific examples to compare and verify the target detection, drivable area segmentation, and lane detection components of the above embodiment. See the description below for details:

[0169] To verify the effectiveness of this invention, the proposed method was compared with other advanced autonomous driving visual perception methods on the large-scale autonomous driving visual perception dataset BDD100K. In the object detection subtask, recall and average precision (mAP50) with an IoU threshold of 0.5 were used as evaluation metrics. For the lane detection subtask, accuracy (Acc) and IoU were used as evaluation metrics. For the drivable area segmentation subtask, average IoU was used to evaluate segmentation performance. Model training used four NVIDIA GeForce GTX 1080Ti processors, with an initial learning rate of 0.001. To accelerate network convergence and improve stability, a pre-heating learning rate adjustment strategy was employed. Adam was used as the model optimizer, with β1 and β2 equal to 0.937 and 0.999, respectively. The batch size was set to 96, and the iteration cycle to 240. Considering both performance and speed, dataset preprocessing was divided into two steps. Regarding resolution adjustment, the image size was adjusted from 1280×720×3 to 640×384×3. For data augmentation, in addition to traditional enhancement methods such as random scaling and cropping, translation and rotation, and adjusting brightness, saturation, and hue, two multi-image weighted stitching algorithms, Mixup and Mosaic, were used to enrich the image background. For model speed testing, a single NVIDIA GeForce GTX 1080Ti image was used, with the speed metric being frames per second (fps). The comparative experimental results for object detection, drivable region segmentation, and lane detection are shown in Tables 1, 2, and 3, respectively. Algorithms marked with an asterisk (*) are multi-task algorithms, capable of simultaneously performing object detection, drivable region segmentation, and lane detection. Algorithms without special markings are single-task algorithms.

[0170] In the object detection subtask, the proposed method of this invention is compared with single-task object detection methods such as Faster-RCNN (Ren SQ, He KM, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.), YOLOv5s (Zhu X, Lyu S, Wang X, et al. TPH-YOLOv5: Improved YOLOv5 based on transformer predictionhead for object detection on drone-captured scenarios[C]. Proceedings of the IEEE / CVF International Conference on Computer Vision, Montreal, BC, Canada: IEEE, 2021: 2778-2788.), and PP-YOLOE (Xu S, Wang X, Lv W, et al. PP-YOLOE: An evolved version of YOLO [EB / OL]. (2022-03-30) [2023-09-05]. https: / / arxiv.org / pdf / 2203.16250.pdf. ), MultiNet (Teichmann M, Weber M, Zoellner M, et al. MultiNet: Real-time joint semantic reasoning for autonomous driving [C]. 2018 IEEE Intelligent Vehicles Symposium, Changshu, Suzhou, China: IEEE, 2018: 1013-1020.), and DLT-Net (Qian Y, Dolan JM, Yang M. DLT-Net: Joint detection of drivable areas, lane lines, and traffic objects [J].A comparison was made between the following libraries: IEEE Transactions on Intelligent Transportation Systems, 2019, 21(11):4670-4679; YOLOP (Wu D, Liao MW, Zhang WT, et al. YOLOP: You only look once forpanoptic driving perception[J]. Machine Intelligence Research, 2022:550-562); and AMTNet (Huang P, Zhu Y. Multi-task data augmentation method joint object detection and semantic segmentation[C]. 2022 International Conference on Machine Learning and Knowledge Engineering, 2022:134-138). The results are shown in Table 1.

[0171] Table 1 Target Detection Results

[0172]

[0173] Analysis of Table 1 shows that, compared with previous advanced methods, the proposed method effectively improves the average detection precision and recall through task feature encoding. Compared with the multi-task methods YOLOP and AMTNet, the precision is improved by 2.7% and 1.4%, respectively, while the recall is improved by 2.5% and 3.4%. Regarding testing speed, the proposed method has a frame processing rate greater than 60 frames / second, meeting the real-time requirements of the network. The single-task method YOLOv5s achieves the best speed performance, but its precision and recall are inferior to the proposed method. Therefore, in the target detection subtask, compared with other methods, the proposed method achieves an effective balance between accuracy and real-time performance.

[0174] In the drivable area segmentation subtask, the proposed method is combined with the single-task drivable area segmentation method PSPNet (reference: Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network [C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA: IEEE, 2017: 6230-6239.), the multi-task visual joint perception method MultiNet (reference: Teichmann M, Weber M, Zoellner M, et al. MultiNet: Real-time jointsemantic reasoning for autonomous driving [C]. 2018 IEEE Intelligent Vehicles Symposium, Changshu, Suzhou, China: IEEE, 2018: 1013-1020.), and DLT-Net (reference: Qian Y, Dolan JM, Yang M. DLT-Net: Joint detection of drivable areas, lane lines, and traffic objects [J]. IEEE Transactions on The results of the comparison between YOLOP (Wu D, Liao MW, Zhang WT, et al. YOLOP: You only look once for panoptic driving perception [J]. Machine Intelligence Research, 2022: 550-562.) and AMTNet (Huang P, Zhu Y. Multi-task dataaugmentation method joint object detection and semantic segmentation [C]. 2022 International Conference on Machine Learning and Knowledge Engineering, 2022: 134-138.) are shown in Table 2.The experimental results show that, compared with other methods in the table, the method proposed in this invention performs best in terms of segmentation accuracy, with an average intersection-union ratio of 92.8%, which is 1.3% and 4.5% higher than the multi-task methods YOLOP and AMTNet, respectively, while still ensuring real-time inference.

[0175] Table 2. Results of Driving Area Segmentation

[0176]

[0177]

[0178] In the lane detection subtask, the proposed method of this invention is compared with single-task lane detection methods ENet (Paszke A, Chaurasia A, Kim S, and et al. ENet: A deep neural network architecture for real-time semantic segmentation [EB / OL]. (2016-06-07) [2023-09-05]. https: / / arxiv.org / pdf / 1606.02147.pdf.), SCNN (Pan X, Shi J, Luo P, et al. Spatial asdeep: Spatial CNN for traffic scene understanding [C]. Proceedings of the AAAIConference on Artificial Intelligence, New Orleans, Louisiana, USA: AAAIPress, 2018(1):7276-7283.) and ENet-SAD (Hou Y, Ma Z, Liu C, et al. Learning lightweight lane detection CNNs by self attention). distillation[C]. Proceedings of the IEEE / CVF International Conference on Computer Vision, Seoul, Korea (South): IEEE, 2019: 1013-1021.), YOLOP (Wu D, Liao MW, Zhang WT, et al. YOLOP: You only look once for panoptic driving perception[J]. Machine Intelligence Research, 2022: 550-562.), AMTNet (Huang P, Zhu Y. Multi-task data augmentation method joint object detection and semanticsegmentation[C].A comparison was made with the findings from the 2022 International Conference on Machine Learning and Knowledge Engineering, 2022: 134-138. Table 3 shows the lane detection results.

[0179] Table 3 Lane line detection results

[0180]

[0181] As shown in Table 3, the method proposed in this invention effectively improves the performance of lane detection based on semantic segmentation. Compared with the multi-task methods YOLOP and AMTNet, the accuracy is improved by 10.6% and 7.5%, respectively. Compared with other methods in the table, the method of this invention achieves the best performance.

[0182] To verify the effectiveness of the method of the present invention in actual traffic applications, three different real road scenarios were selected to compare the multi-task visual driving joint perception method YOLOP, which does not adopt the task feature decoupling method, with the method proposed in this invention. Figure 7 From left to right, the images show a visual comparison of the two methods under three different scenarios: daytime, dusk, and nighttime. To more intuitively demonstrate the detection performance of the two methods, a combination of bounding boxes and classification confidence scores is used for prediction. Figure 7 (a) Figure 7 (b) shows the visualization results of YOLOP and the method of this invention, respectively. The target category and classification confidence score are marked above the bounding box. The first digit "0" indicates that the detected target category is a car, and the decimal part represents the classification confidence score corresponding to the target.

[0183] By comparing the visualization results of the YOLOP multi-task perception method and the method of this invention in the first column of daytime scenes, it can be seen that when lane lines or drivable areas are obscured by large vehicles, YOLOP suffers from problems such as discontinuous lane lines and incorrect drivable area segmentation. Figure 7 As shown in the elliptical box (a). Compared to the YOLOP method, the proposed method can make smoother and more accurate segmentation predictions. However, in terms of object detection, the multi-task method YOLOP exhibited false negatives, failing to successfully identify any cars marked in the ellipse, while the proposed method effectively identified vehicles ahead with a classification confidence score of 0.96. In the second column of the twilight scene, the multi-task perception method YOLOP made incorrect lane line edge segmentation predictions, such as... Figure 7As shown in the ellipse in (a), the method of this invention effectively alleviates the problem of discontinuous and unclear edge segmentation, indirectly confirming the effectiveness of the dual-resolution segmentation decoupling branch. In the third column, a nighttime scene, a comparison of the visualization results of the two methods shows that the lane line detection results obtained by the multi-task method YOLOP have discontinuities, while the method of this invention can guarantee relatively complete and smooth pixel-level prediction, demonstrating better performance in edge segmentation and internal consistency. By comparing the detection results of the two methods in three different scenarios, it can be seen that the method of this invention can effectively improve the classification confidence of correctly located targets, and to a certain extent deepen the correlation between classification confidence and localization branch, further illustrating the effectiveness of the object detection classification and localization separation coding.

[0184] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. Other embodiments can be used by those skilled in the art when reading the above description. Furthermore, in the above detailed description, various features may be grouped together to simplify the invention. This should not be construed as an intention that a feature of an unclaimed invention is necessary for any claim. Rather, the subject matter of the invention may be less than all the features of a particular embodiment of the invention. Thus, the following claims are incorporated herein by reference as examples or embodiments, wherein each claim is an independent, separate embodiment, and these embodiments are contemplated to be combined with each other in various combinations or arrangements. The scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A method for joint vision perception in autonomous driving, characterized in that, The method includes: Based on traffic road images, different levels of backbone features are obtained. These different levels of backbone features are aggregated to obtain multi-scale features. Based on the weighting coefficients of each level of features obtained by dynamic convolutional adaptation, the multi-scale features are weighted and summed to obtain multi-scale fusion coding features. An object detection decoding network is constructed, which includes a classification branch and a localization branch. The object detection decoding network takes the multi-scale fusion coding features as input and generates context coding features and multi-level spatial texture detail coding features based on the multi-scale fusion coding features. The context coding features and multi-level spatial texture detail coding features are respectively input to the classification branch and the localization branch for prediction to obtain classification features and localization features. An IoU perception prediction branch is used for auxiliary learning. A dual-branch semantic segmentation decoding network is constructed. The dual-branch semantic segmentation decoding network uses the low-resolution features and high-resolution features in the multi-scale fusion coding features as inputs and feeds them into the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are obtained through the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are fused to obtain the target features. The dual-branch semantic segmentation decoding network is trained and learned using the edge loss function and the overall segmentation loss function.

2. The method according to claim 1, characterized in that, The context-encoded features are generated using the following method: Obtaining multi-scale fusion coding features F l F l+1 , l represents the feature level at which the fused feature is located, where F l For a fusion feature that is 1 / 16 the resolution of the original image, F l+1 The fusion feature is 1 / 32 of the original image resolution; right F l+1 The downsampling feature F is obtained by performing downsampling. l+2 , where F l+2 The fusion feature is 1 / 64 the resolution of the original image; Will F l F l+1 F l+2 As multi-scale feature input, semantic context feature encoding at the l+n pyramid level is achieved, resulting in context-encoded feature F. l ' +n : Where n = [0,1,2], CBS represents the processing procedure performed by a convolutional normalization activation function combination module consisting of a 1×1 convolutional layer with a stride of 1, a batch normalization layer, and an activation function layer, f down This indicates a 1×1 convolutional downsampling layer with a stride of 2.

3. The method according to claim 2, characterized in that, The classification features are obtained using the following method: The classification branch f consists of two convolutional layers. cls (·) Predict the classification score, and after feature recombination, obtain the classification feature with size B×H×W×C, where B is the batch number, H and W are the height and width of the feature map respectively, and C is the number of categories.

4. The method according to claim 1, characterized in that, The multi-level spatial texture detail encoding features are generated using the following method: Obtaining multi-scale fusion coding features F l-1 F l F l+1 , l represents the feature level at which the fused feature is located, where F l-1 For a fusion feature that is 1 / 8 the resolution of the original image, F l For a fusion feature that is 1 / 16 the resolution of the original image, F l+1 The fusion feature is 1 / 32 of the original image resolution; The multi-level spatial texture detail encoding feature F is calculated using the following formula. l ' +m : Where m = [-1, 0, 1], Concat represents the feature concatenation operation, CBS represents the processing procedure performed by a convolutional normalization activation function combination module consisting of a 1×1 convolutional layer with a stride of 1, a batch normalization layer, and an activation function layer, and f down f represents a 1×1 convolutional downsampling layer with a stride of 2. up This is a bilinear interpolation upsampling operator with an upsampling factor of 2.

5. The method according to claim 4, characterized in that, The localization branch includes a regression branch f. reg (·), Target confidence branch f obj (·) and IoU perception prediction branch f IoU (·), the localization features are obtained through the following method: The multi-level spatial texture detail encoding features are input into the localization branch to predict the target bounding box. After feature recombination, coordinate predictions of size B×H×W×4, confidence predictions of size B×H×W×1, and intersection-union ratio (IoU) score predictions of size B×H×W×1 are obtained, where B is the batch number, and H and W are the height and width dimensions of the feature map, respectively.

6. The method according to claim 5, characterized in that, The method of using the IoU-aware prediction branch for assisted learning includes: During the training phase, the IoU-aware prediction branch uses the binary cross-entropy function to calculate the loss L for predicting positive samples. IoU : Where N is the number of positive samples, s is any number of positive samples, pos is the set of positive samples, BCE(·) is the binary cross-entropy function, and for each positive sample, the predicted IoU score obtained by the IoU-aware prediction branch is expressed as: For the corresponding prediction box With real frame The true value of the IoU score is calculated using the overlap(·) function; During the model inference phase, the original classification score, target confidence score, and IoU perceptual prediction score are multiplied to obtain the final confidence score S. fin : For each positive sample s, the corresponding classification score, target confidence score, and IoU perception prediction branch prediction score are cls, respectively. s obj s IoU s The hyperparameter λ takes values ​​in the range (0,1).

7. The method according to claim 1, characterized in that, Using the low-resolution and high-resolution features from the multi-scale fusion coding features as input, and feeding them into the low-resolution branch and high-resolution branch respectively, low-frequency main features and high-frequency edge features are obtained through the low-resolution branch and high-resolution branch respectively. The low-frequency main features and high-frequency edge features are then fused to obtain the target features, including: Convolution and bilinear interpolation upsampling are performed on low-resolution and high-resolution features respectively to obtain intermediate feature maps. The low-resolution and high-resolution features are then concatenated with the intermediate feature maps to obtain an image frame sequence. The image frame sequence is processed using an optical flow field migration operator to obtain the motion offset of each pixel. An optical flow strain feature map is determined based on the motion offset. The pixel value of each pixel in the optical flow strain feature map is half of the sum of the original pixel value and the pixel value corresponding to its motion offset. The optical flow strain feature map is stitched together with low-resolution and high-resolution features respectively. Differentiable bilinear interpolation is performed on multiple neighboring pixels around the feature stitching result by optical flow field distortion operator, and the corresponding pixel values ​​of multiple adjacent pixels are weighted and summed to obtain low-frequency main features. The low-frequency main features include low-resolution low-frequency main features and high-resolution low-frequency main features. The high-frequency edge features include low-resolution high-frequency edge features and high-resolution high-frequency edge features. The low-resolution features and high-resolution features are subtracted pixel by pixel from the corresponding low-resolution low-frequency main features and high-resolution low-frequency main features to obtain the low-resolution high-frequency edge features and high-resolution high-frequency edge features. In the direction from high-resolution branch to low-resolution branch, two convolution operators are used to downsample the high-resolution high-frequency edge features. After adding them with the low-resolution high-frequency edge features, a bilinear interpolation upsampling operation is used to upsample the sum, resulting in the downsampled matching feature output in the corresponding direction. In the direction from low-resolution branch to high-resolution branch, a bilinear interpolation upsampling operation is used to upsample the low-resolution high-frequency edge features. The feature weight map is obtained through an activation function and multiplied with the high-resolution high-frequency edge features in a weighted manner to obtain the upsampled matching feature output in the corresponding direction. The downsampled matching feature output and the upsampled matching feature output are fused using a feature concatenation method. After continuous upsampling operations, the final high-frequency edge features are obtained. In the direction from high-resolution branch to low-resolution branch, two convolution operators are used to downsample the high-resolution low-frequency main features. After adding them with the low-resolution low-frequency main features, a bilinear interpolation upsampling operation is used to upsample the sum, resulting in the downsampled matching feature output in the corresponding direction. In the direction from low-resolution branch to high-resolution branch, a bilinear interpolation upsampling operation is used to upsample the low-resolution low-frequency main features. The feature weight map is obtained through an activation function and multiplied with the high-resolution low-frequency main features in a weighted manner to obtain the upsampled matching feature output in the corresponding direction. The downsampled matching feature output and the upsampled matching feature output are fused using a feature concatenation method. After continuous upsampling operations, the final low-frequency main features are obtained. The final high-frequency edge features and the final low-frequency main features are added pixel by pixel to obtain the target features.

8. The method according to claim 7, characterized in that, The dual-branch semantic segmentation and decoding network is trained using edge loss functions and overall segmentation loss functions, including: The loss function is expressed as: Among them, L total L full L edge These are the total loss function, the overall segmentation loss function, and the marginal loss function, respectively. The overall segmentation loss function uses the binary cross-entropy function, F. f ' ull Indicate target features, For the corresponding complete truth labeling graph, F e ' dge Indicates the final high-frequency edge features, The corresponding edge truth label map is calculated using the following formula: Where p is any pixel value in the edge label matrix. For the complete truth label graph edge nearest neighbor set, δ is the Euclidean distance between the pixel value and the set of nearest neighbors on the edge of the complete ground truth label image. δ is a set threshold. When the Euclidean distance is not greater than δ, the value of the corresponding pixel p is set to 1. When the Euclidean distance is less than δ, the value of the corresponding pixel p is set to 0. Edge loss function L edge The calculation formula is as follows: Where I represents the image input, p i The edge truth label representing pixel i, q i This represents the edge prediction result for pixel i.

9. An apparatus for a vision-based joint perception method for autonomous driving, characterized in that, The device includes: The fusion feature calculation module is configured to obtain backbone features at different levels based on traffic road images, aggregate the backbone features at different levels to obtain multi-scale features, and perform weighted summation on the multi-scale features based on the weighting coefficients of each level of features obtained by dynamic convolutional adaptation to obtain multi-scale fusion coding features. The target category location recognition module is configured to construct a target detection decoding network, which includes a classification branch and a localization branch. The target detection decoding network takes the multi-scale fusion coding features as input and generates context coding features and multi-level spatial texture detail coding features based on the multi-scale fusion coding features. The context coding features and multi-level spatial texture detail coding features are respectively input to the classification branch and the localization branch for prediction to obtain classification features and localization features. An IoU perception prediction branch is used for auxiliary learning. The drivable area and lane line recognition module is configured to construct a dual-branch semantic segmentation and decoding network. The dual-branch semantic segmentation and decoding network uses the low-resolution and high-resolution features in the multi-scale fusion coding features as inputs and feeds them into the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are obtained through the low-resolution branch and the high-resolution branch respectively. The low-frequency main features and high-frequency edge features are fused to obtain the target features. The edge loss function and the overall segmentation loss function are used to guide the dual-branch semantic segmentation and decoding network to train and learn.

10. A readable storage medium, characterized in that, The readable storage medium stores one or more programs, which can be executed by one or more processors to implement the method as described in any one of claims 1 to 8.

Images