Traffic scene oriented scalable multi-task visual perception method

Abstract

The application discloses a kind of traffic scene-oriented scalable multi-task visual perception method, comprising the following steps: design multi-task visual perception model overall architecture;Build main feature extraction network Build task branch network;Network training;Detect multi-task network.The application separates the method of main feature extraction network and branch network training and saves weight, improves the real-time performance of entire model in traffic scene multi-task detection, while taking into account the scalability of model.The application designs a convolution multi-channel attention residual module by increasing side road weight branch network, which can not only alleviate the problem caused by scale change, but also make the network pay more attention to the objects to be detected, effectively ignoring the interference of background.The application not only alleviates the problem of occupying a large amount of hardware resources for visual multi-task detection, improves the real-time performance of the network, but also improves the scalability of the network, so that it can adapt to new task requirements.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent transportation, and particularly relates to an extensible multi-task visual perception method for a traffic scene. BACKGROUND

[0002] Multi-task vision is an important basis for realizing automatic driving in a complex traffic scene. Through target detection, drivable area segmentation, human pose estimation and other visual tasks by a deep neural network, important basis can be provided for path planning of the vehicle.

[0003] At present, there are two kinds of multi-task visual detection methods for a traffic scene: one is to build a deep neural network for each specific visual task, and then parallel all the detection networks on a vehicle terminal to realize multi-task detection; the second method is that the whole multi-task detection network shares a main feature extraction network, and then connects multiple specific task branch networks. In the first method for realizing multi-task detection, each specific task is calculated separately, which can better extend new visual tasks, but due to parallel operation of all networks, it will occupy a large memory and video memory, reduce the detection speed, and has poor real-time performance. When the second method is used, a feature extraction network is shared, which reduces the parameter quantity and calculation quantity of the network, but the loss function of the network is set in advance according to the existing tasks, multiple tasks are coupled together, and new visual tasks need to retrain the network, which cannot better extend new visual tasks. Therefore, it is necessary to study a multi-task extensible visual perception architecture which shares a main network branch network but does not need to retrain the whole network when a new visual task is added. The above two methods can realize multi-task detection, but each has certain limitations. SUMMARY

[0004] In order to solve the above problems in the prior art, the purpose of the present application is to provide an extensible multi-task visual perception method for a traffic scene, which can improve the real-time performance of multi-task detection and the expandability of the detection process.

[0005] In order to achieve the above purpose, the technical scheme of the present application is as follows: an extensible multi-task visual perception method for a traffic scene, comprising the following steps:

[0006] A, designing a multi-task visual perception model overall architecture

[0007] The overall architecture of the multi-task visual perception model is composed of a backbone feature extraction network and a task branch network. The backbone feature extraction network is stacked by a series of convolutional layers and multi-channel attention residual modules. After building the backbone feature extraction network, the parameters in the backbone feature extraction network are adjusted, and the adjusted parameters are saved. Then the output of the last three layers of the backbone feature extraction network is input into the candidate box generation network, and after processing by the candidate box mapping module, it is input into the task branch network at the same time. The task branch network includes a target detection branch network, a human key point detection branch network, a drivable area segmentation branch network, and a lane line detection branch network. The lane line detection branch network and the drivable area branch network are merged into one task branch network, and the drivable area segmentation branch network simultaneously realizes the functions of the drivable area segmentation branch network and the lane line detection branch network. The target detection branch network is composed of convolutional layers, pooling layers and batch normalization layers in series. The human key point detection branch network, the drivable area segmentation branch and the lane line detection branch network are composed of a series of deconvolutional layers in series. After building the task branch network, the task branch network is trained respectively, and the trained task branch network parameters are saved. Finally, the parameters of the backbone feature extraction network and the parameters of the task branch network are loaded to predict the image obtained by the vehicle-mounted camera. The specific steps are as follows:

[0008] A1, input the image obtained by the vehicle-mounted camera into the backbone feature extraction network to obtain a feature map.

[0009] A2, the obtained feature map is sent to the candidate box generation network, and after obtaining the candidate box, the candidate box is mapped to the down-sampled feature map through the candidate box mapping module.

[0010] A3, the feature map after candidate box mapping is sent to the target detection branch network and the human key point detection branch network in parallel.

[0011] A4, when performing human key point detection branch network detection, the target frame obtained by target detection is coupled with the candidate region of human key point detection, so that the human key point detection branch network has a region of interest.

[0012] A5, the feature map obtained by the backbone feature extraction network is mapped through the candidate box and then input into the drivable area segmentation branch network and the lane line detection branch network to up-sample the feature map, obtain the segmentation of the drivable area and the detection result of the lane line, and mark the final result on the image obtained by the vehicle-mounted camera at the beginning.

[0013] B, build the backbone feature extraction network

[0014] The backbone feature extraction network draws lessons from a pure convolutional network structure, designs a convolutional multi-channel attention residual module, i.e., a Conv-MA-Next Block, stacks the convolutional multi-channel attention residual module to form the backbone feature extraction network, and the specific steps are as follows:

[0015] B1, first use The image obtained by the vehicle-mounted camera is down-sampled to obtain a feature map, and after down-sampling, the down-sampled feature map is normalized by using a layer normalization technology.

[0016] B2, build a convolutional multi-channel attention residual module. The convolutional multi-channel attention residual module adopts The network parameter quantity. The convolutional multi-channel attention residual module branch network is represented as:

[0017] G(x) = Sigmoid(BN(Conv(GE(BN(Conv(GE(g(x))))))))

[0018] In the formula, x represents an input feature map.

[0019] G(x) represents the channel weight obtained by passing through the attention branch network.

[0020] Sigmoid represents an activation function, and the function expression is .

[0021] BN represents a batch normalization layer, which converts the data input into the BN layer into a distribution with a mean of 0 and a variance of 1.

[0022] Conv represents a convolution operation, which means multiplying a specified size matrix data with the input data.

[0023] GE represents a GELU activation function, and the function expression of the GELU activation function is GELU(x)=x∙Φ(x), Φ(x) represents a cumulative probability distribution of a Gaussian distribution, that is, the definite integral of the Gaussian distribution in the interval (- ,x] interval.

[0024] g(x) represents global pooling.

[0025] Finally, the obtained channel weight G(x) is multiplied with the original feature map to make the backbone feature extraction network pay more attention to the target region. The formula of the convolutional multi-channel attention residual module is:

[0026] C(x) = G(x) dropout(Lscale(Conv(GE(Conv(BN(dwConv(x)))))))

[0027] where dropout represents randomly turning off some channels to make them lose their functions.

[0028] Lscale table is channel scaling, which reduces the number of channels of the feature map by a certain ratio.

[0029] dwConv represents a depth separable convolution, which convolves the input data by dimension.

[0030] B3, the built convolution multi-channel attention residual module is stacked in different dimensions by a certain ratio. The specific formula is as follows:

[0031] output = a C(b C(c C(d C(x))))

[0032] where C represents a convolution multi-channel attention residual module.

[0033] a, b, c, d are all integers, representing different ratios.

[0034] Output represents output.

[0035] C, build task branch network

[0036] The target detection branch network, the human key point detection branch network and the drivable area segmentation branch network are built to realize the functions of recognizing pedestrians and vehicles in the traffic scene, detecting human key points, segmenting drivable areas and detecting lane lines. The specific steps are as follows:

[0037] C1, build target detection branch network

[0038] The target detection branch network is composed of two fully connected layers, which are divided into class prediction and target detection frame regression after the fully connected layer. The loss function of the target detection branch network is:

[0039]

[0040] where, represents the probability that the i-th candidate frame is the real label. i

[0041] represents 1 when the sample in the candidate frame is a positive sample, and 0 when it is a negative sample.

[0042] represents the boundary frame regression parameter of the i-th candidate frame. i

[0043] represents the​​i the real box parameters corresponding to the candidate box.

[0044] represents the number of samples of one training input.

[0045] represents the number of candidate box positions.

[0046] represents the classification loss, and the binary cross-entropy loss is as follows:

[0047]

[0048] represents the regression loss, and the formula is as follows:

[0049]

[0050] wherein the function is as follows:

[0051]

[0052] C2, building a human key point detection branch network

[0053] The human key point detection branch network first uses a candidate region mapping module for mapping, extracts features of the feature map obtained by mapping through multiple convolutions and activation function activation, and finally performs transposed convolution on the feature map activated by the convolution and the activation function to expand the height and width of the feature map, so that the height and width of the feature map are equal to the height and width of the image obtained by the vehicle-mounted camera at the beginning, the number of channels of the feature map is the same as the number of key points, and each channel predicts a key point.

[0054] C3, building a drivable area segmentation branch network and a lane line detection branch network

[0055] The drivable area segmentation branch network and the lane line detection branch network have the same structure, and are combined in a drivable area segmentation branch network for detection. The drivable area segmentation branch network is built using a candidate region mapping module for mapping, converting the size of the feature map and performing convolution on it. Finally, the feature map after convolution is transposed to expand the height and width of the feature map, so that the height and width of the feature map are equal to the height and width of the image obtained by the vehicle-mounted camera at the beginning, the number of channels of the feature map is the same as the number of categories, and each channel predicts a category. One of the channels is used as the output of the lane line detection branch network for lane line detection.

[0056] E, training the network

[0057] D1, training the backbone feature extraction network

[0058] The backbone feature extraction network is trained jointly with the target detection branch network. After inputting the original data, the parameters of the backbone feature extraction network are initialized, the feature map obtained is input into the target detection branch network, the loss function is calculated by comparing with the sample label, the gradient descent method is used to update the parameters to reduce the loss, and when the difference between the output value of the loss function and the output value of the last round is less than , the training is stopped, and the weights of the backbone feature extraction network and the weights of the target detection branch network are saved respectively.

[0059] D2, training task branch network

[0060] The original data is input into the backbone feature extraction network with loaded weights, as data preprocessing. The parameters of the task branch network are initialized, the feature map obtained by preprocessing is input into the task branch network, the loss is calculated by comparing with the label, the parameters are updated by the gradient descent method, and the loss value is reduced. When the difference between the output value of the loss function and the output value of the last round is less than , the training is stopped, and the task branch network weight is saved.

[0061] E, detecting multi-task network

[0062] E1, loading the weights of the backbone feature extraction network , to obtain the backbone feature extraction network Conv-Next-Net( ) with specific parameters.

[0063] E2, loading the weights of the target detection branch network , to obtain the target detection branch network Det( ) with specific parameters.

[0064] E3, loading the weights of the human key point detection branch network , to obtain the human key point detection branch network K( ) with specific parameters.

[0065] E4, loading the weights of the drivable area segmentation branch network , to obtain the drivable area segmentation branch network SR( ) with specific parameters.

[0066] E5, inputting the image to be detected into the backbone feature extraction network to obtain the backbone extraction feature as follows:

[0067] Conv-Next-Net( )[input]

[0068] input represents the image obtained by the vehicle-mounted camera.

[0069] represent the obtained feature map.

[0070] E6, the obtained feature map is input into each task branch network to obtain the following output:

[0071]

[0072] represent the final output.

[0073] represent the new task branch network loaded with the weight.

[0074] E7, the obtained output The corresponding target position, drivable area segmentation area and human body key point are marked in the initially obtained camera input image to obtain a final detection map.

[0075] Compared with the prior art, the present application has the following beneficial effects:

[0076] 1, the present application designs a method different from end-to-end training, which separates the backbone feature extraction network and the branch network for training and saving the weight, improves the real-time performance of the whole model in the multi-task detection of the traffic scene, and takes into account the scalability of the model, separates the weight of the backbone feature extraction network and the weight of the task branch network, which can be more convenient to train the task branch network separately and expand.

[0077] 2, the present application is based on the ConvNext backbone network, and a convolution multi-channel attention residual module (Conv-MA-Block Block) is designed by increasing the side weight branch network, which can not only alleviate the problem caused by the change of scale, but also make the network pay more attention to the object to be detected, and can effectively ignore the interference of the background. Because there is some correlation between different visual tasks, the addition of the extraction of the candidate region frame can greatly improve the detection accuracy of target detection and human body key points. Therefore, the present application not only alleviates the problem that visual multi-task detection needs to occupy a large amount of hardware resources, improves the real-time performance of the network, but also improves the scalability of the network, so that it can adapt to new task requirements. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 is the training and detection process diagram of the multi-task detection network of the present application.

[0079] Figure 2 is the overall framework structure diagram of the present application.

[0080] Figure 3 is the backbone feature extraction network structure diagram of the multi-task network.

[0081] Figure 4 A structure diagram of a branch network for a target detection task of the application.

[0082] Figure 5 A structure diagram of a branch network for human key point detection of the application.

[0083] Figure 6 A structure diagram of a branch network for drivable area segmentation of the application.

[0084] Figure 7 A flow chart of a training process of the backbone feature extraction network.

[0085] Figure 8 A flow chart of a training process of the task branch network. DETAILED DESCRIPTION

[0086] To make the purpose, technical solutions and advantages of the application clearer, the embodiments of the application will be further described in detail below with reference to the drawings. As shown in the drawings, the process of the application is as follows: Figure 1

[0087] Step one, as shown in the drawings, the overall framework of the model is designed. The input of the model is the image obtained by the vehicle-mounted camera. The obtained image is first down-sampled to extract features by the backbone feature extraction network to obtain a feature map, and the obtained feature map is sent to the candidate box generation network to obtain a candidate box. After that, the candidate box is mapped to the down-sampled feature map through the candidate box mapping module. Then the feature map with the candidate box is input to the target detection branch network, the human key point branch network, the drivable area segmentation network and the lane line detection branch network to predict the vehicle, pedestrian, road and lane line in the image. Figure 2 Step two, as shown in the drawings, the backbone feature extraction network structure diagram of the multi-task network is used to build the backbone feature extraction network of the model. The input is a three-channel image, and the size of the image is 224

[0088] 224. First, it is down-sampled by a layer of convolution, the size of the convolution kernel is 4 Figure 3 4, the step is 4, and the size of the down-sampled image is 56 56, the number of channels is increased to 96. After a layer normalization (Layer Norm) operation, the image is input to the convolution multi-channel attention residual module (Conv-MA-Block Block). The specific details of the construction of the convolution multi-channel attention residual module are shown in the dashed box in the drawings, and the overall structure is composed of two parallel branch networks. The feature extraction branch network is first passed through a 7

[0089] x 7 convolution kernel, and the size of the down-sampled image is 56 Figure 3 56. The number of channels is increased to 96. After a layer normalization (Layer Norm) operation, the image is input to the convolution multi-channel attention residual module (Conv-MA-Block Block). ​​7, stride is 1, padding is 3, the feature map size is unchanged, and a layer normalization is connected after the convolution layer; continue to perform convolution, the convolution kernel size is 1, the stride is 1, and the channel number is deepened to four times of the original after the convolution is activated by a GELU activation function; repeat the convolution and scale the channel to prevent overfitting. The rightmost parallel branch in the figure is first globally pooled, then activated by a GELU activation function, then a series of convolution and normalization processing are performed, and finally a Sigmoid function is used for normalization to obtain the weight of each channel in the feature map.

[0090] The convolution multi-channel attention residual module can be represented as

[0091] G(x) = Sigmoid(BN(Conv(GE(BN(Conv(GE(g(x))))))))

[0092] x represents an input feature map.

[0093] G(x) represents the channel weight obtained through the attention branch network.

[0094] Sigmoid represents an activation function, and the function expression is .

[0095] BN represents a batch normalization layer.

[0096] Conv represents convolution.

[0097] GE represents an activation function named GELU.

[0098] g(x) represents global pooling.

[0099] The obtained channel weight is multiplied by the feature extraction branch network to obtain the final feature map.

[0100] The convolution multi-channel attention residual module is stacked in the number of [3, 3, 9, 3], the down-sampling dimension of the first stacked module is 96, the down-sampling dimension of the second module is 192, the down-sampling dimension of the third module is 384, and the down-sampling dimension of the fourth module is 768. The feature map size is aligned between each module through down-sampling (Downsample). Down-sampling (Downsample) is composed of a layer normalization and a convolution layer with a kernel size of 2 and a stride of 2. After the main feature extraction network, the size of the input image changes from the original 224 224 3 to 7 7 768.

[0101] Step three, as shown in Figure 4 The target detection branch network structure diagram is built as shown in the target detection branch network.

[0102] The target detection branch network is mainly composed of two fully connected layers, and is divided into two parts after the fully connected layer, which are class prediction and target detection frame regression. The loss function of the target detection branch network is

[0103]

[0104] The probability that the i-th candidate frame is predicted as a real label is represented.

[0105] The sample in the candidate frame is 1 when it is a positive sample, and 0 when it is a negative sample.

[0106] The boundary frame regression parameter of the i-th candidate frame is represented.

[0107] The real frame parameter corresponding to the i-th candidate frame is represented.

[0108] The number of all samples in a small batch is represented.

[0109] The number of candidate frame positions is represented.

[0110] The classification loss adopts binary cross entropy loss ,

[0111] The regression loss ,

[0112] Wherein The function is

[0113] Step four, as shown in Figure 6 The drivable area segmentation branch network structure diagram is built as shown in the drivable area segmentation branch network.

[0114] The drivable area segmentation branch network uses the candidate region mapping module (RoIAlign) for mapping, converts the feature map size to 7 7, and then performs convolution, and finally performs 4 times of transposed convolution for upsampling, so that the feature map channel number is the same as the class number, and each channel predicts a class mask.

[0115] Step five, as shown in Figure 5 The pedestrian key point detection branch network structure diagram is built as shown in the pedestrian key point detection branch network.

[0116] The human key point detection branch network first uses a candidate region mapping module (RoIAlign) to map, converts the feature map size to 14 14, and then passes through 4 convolutions with a convolution kernel size of 3 3, and finally performs transpose convolution on the feature map to make the feature map channel number the same as the key point number, and each channel predicts a key point.

[0117] Step six, the backbone feature extraction network and the task branch network are trained respectively.

[0118] The training process of the backbone feature extraction network is shown in Figure 7 . The backbone feature extraction network is jointly trained with the target detection branch network, and the data set used is BDD100K. After inputting the original data, the backbone feature extraction network parameters are initialized, the obtained feature map is input into the target detection branch network, the loss function is calculated by comparing with the sample label, the gradient descent method is used to update the parameters to reduce the loss, and when the difference between the output value of the loss function and the output value of the last round is less than , the weights of the backbone feature extraction network are saved respectively.

[0119] The training process of the task branch network is shown in Figure 8 . The original data is input into the backbone feature extraction network with loaded weights as data preprocessing. The task branch network parameters are initialized, the feature map obtained by preprocessing is input into the branch network, the loss is calculated by comparing with the label, the parameters are updated by the gradient descent method to reduce the loss value, and when the difference between the output value of the loss function and the output value of the last round is less than , the task branch network weights are saved.

[0120] The above describes the basic principles, main features and advantages of the present application. It should be understood by those skilled in the art that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application.

Claims

1. A scalable multi-task visual perception method for traffic scenarios, characterized in that: Includes the following steps: A. Design the overall architecture of a multi-task visual perception model The overall architecture of the multi-task visual perception model consists of a backbone feature extraction network and task branch networks. The backbone feature extraction network is composed of a series of convolutional layers and multi-channel attention residual modules. After the backbone feature extraction network is built, it is trained, and the parameters in the backbone feature extraction network are adjusted and saved. Then, the outputs of the last three layers of the backbone feature extraction network are input into the candidate box generation network. After being processed by the candidate box mapping module, they are simultaneously input into the task branch networks. The task branch networks include an object detection branch network, a human keypoint detection branch network, a drivable region segmentation branch network, and a lane line detection branch network, among which lane line detection... The drivable region branch network and the drivable region branch network are merged into a single task branch network. The drivable region segmentation branch network simultaneously performs the functions of both the drivable region segmentation branch network and the lane detection branch network. The object detection branch network consists of concatenated convolutional layers, pooling layers, and batch normalization layers. The human keypoint detection branch network, the drivable region segmentation branch network, and the lane detection branch network are all composed of a series of deconvolutional layers. After constructing the task branch networks, they are trained separately. The trained task branch network parameters are saved. Finally, the images acquired by the vehicle camera are predicted by loading the parameters of the backbone feature extraction network and the task branch networks. The specific steps are as follows: A1. Input the image acquired by the vehicle-mounted camera into the backbone feature extraction network to obtain the feature map; A2. The obtained feature map is fed into the candidate box generation network. After obtaining the candidate boxes, the candidate box mapping module maps the candidate boxes onto the downsampled feature map. A3. The feature maps after the candidate boxes are mapped are fed into the target detection branch network and the human key point detection branch network in parallel. A4. When performing the detection of the human keypoint detection branch network, the target bounding box obtained by the target detection is coupled with the candidate region of the human keypoint detection, so that the human body region is the region of interest of the human keypoint detection branch network. A5. The feature map obtained by the backbone feature extraction network is mapped through the candidate box and then entered into the drivable area segmentation branch network and the lane line detection branch network to upsample the feature map, so as to obtain the results of drivable area segmentation and lane line detection, and the final results are marked on the image obtained by the vehicle camera at the beginning. B. Constructing the backbone feature extraction network The backbone feature extraction network borrows from the pure convolutional network structure and designs a convolutional multi-channel attention residual module, namely the Conv-MA-Next Block. These convolutional multi-channel attention residual modules are stacked to form the backbone feature extraction network. The specific steps are as follows: B1, First use The image acquired by the vehicle camera is downsampled to obtain a feature map. After downsampling, the downsampled feature map is normalized using layer normalization technology. B2. Construct a convolutional multi-channel attention residual module; the convolutional multi-channel attention residual module adopts... Network parameter count; the convolutional multichannel attention residual module branch network is represented as: G(x) = Sigmoid(BN(Conv(GE(BN(Conv(GE(g(x)))))))) In the formula, x represents the input feature map; G(x) represents the channel weights obtained through the attention branch network; Sigmoid represents the activation function, and its function expression is: ; BN stands for Batch Normalization Layer, which transforms the data input to the BN layer into a distribution with a mean of 0 and a variance of 1. Conv stands for convolution operation, which means multiplying the input data by a matrix of a specified size; GE represents the GELU activation function, and its functional expression is GELU(x) = x∙Φ(x), where Φ(x) represents the cumulative probability distribution of the Gaussian distribution, i.e., in (- The definite integral of the interval [x, y] with respect to the Gaussian distribution; g(x) represents global pooling; Finally, the obtained channel weights G(x) are multiplied by the original feature map to make the backbone feature extraction network focus more on the target region; the formula for the convolutional multi-channel attention residual module is expressed as: C(x) = G(x) dropout(Lscale(Conv(GE(Conv(BN(dwConv(x))))))) In the formula, dropout means randomly shutting down some channels, rendering them ineffective; The Lscale table is for channel scaling, which reduces the number of channels in the feature map by a certain ratio; dwConv stands for depthwise separable convolution, which convolves the input data according to its dimensions; B3. Stack the completed convolutional multi-channel attention residual modules proportionally along different dimensions; the specific formula is as follows: output = a C(b C(c C(d C(x)))) In the formula, C represents the convolutional multichannel attention residual module; a, b, c, and d are all integers, representing different proportions; Output indicates output; C. Build a task branch network An object detection branch network, a human key point detection branch network, and a drivable area segmentation branch network are constructed to achieve the functions of pedestrian and vehicle recognition, human key point detection, drivable area segmentation, and lane line detection in traffic scenes, respectively; the specific steps are as follows: C1. Constructing the target detection branch network The object detection subnetwork consists of two fully connected layers, followed by class prediction and object bounding box regression. The loss function of the object detection subnetwork is: In the formula, Indicates the first i The probability that each candidate box is predicted to be the true label; A value of 1 indicates that the sample within the candidate box is a positive sample, and a value of 0 indicates that the sample is a negative sample. Indicates the prediction of the first i Bounding box regression parameters for each candidate box; Indicates the first i The parameters of the ground truth boxes corresponding to each candidate box; This indicates the number of samples input for one training iteration; Indicates the number of candidate box positions; The classification loss is represented by the binary cross-entropy loss as shown in the following formula: The regression loss is expressed by the following formula: in The function, with the following formula: C2. Construct a human keypoint detection branch network The human keypoint detection branch network first uses the candidate region mapping module to map the feature map. The feature map obtained by mapping is activated by multiple convolutions and activation functions to extract its features. Finally, the feature map after convolution and activation functions is transposed and convolved to expand the height and width of the feature map so that the height and width of the feature map are equal to the height and width of the image initially obtained by the vehicle camera. The number of channels of the feature map is the same as the number of keypoints, and each channel predicts one keypoint. C3. Construct a drivable area segmentation branch network and a lane line detection branch network. The drivable region segmentation branch network and the lane line detection branch network have the same structure, and they are merged into a single drivable region segmentation branch network for detection. This branch network is constructed using a candidate region mapping module to map and transform the feature map size. The feature map is then convolved, and finally transposed and convolved to enlarge its height and width, making them equal to the height and width of the image initially acquired by the vehicle camera. The number of feature map channels is the same as the number of categories, and each channel corresponds to predicting one category. One of the channels is used as the output of the lane detection branch network for lane detection. D. Training Network D1. Training the backbone feature extraction network The backbone feature extraction network and the object detection branch network are jointly trained. After inputting the original data, the parameters of the backbone feature extraction network are initialized. The obtained feature maps are then input into the object detection branch network, and the loss function is calculated by comparing them with the sample labels. Gradient descent is used to update the parameters and reduce the loss. When the difference between the output value of the loss function and the output value of the previous round is less than 1%, the loss is calculated. At this point, training is no longer performed, and the weights of the main feature extraction network and the object detection branch network are saved separately. D2, Training Task Branch Network The raw data is input into the pre-loaded weighted backbone feature extraction network as preprocessing. The parameters of the task branch network are initialized, and the preprocessed feature maps are input into the task branch network. The loss is calculated by comparing the preprocessed feature maps with the labels. The parameters are then updated using gradient descent to reduce the loss value. The loss function is activated when the difference between its output and the previous output is less than a certain threshold. At this point, training is stopped, and the network weights of the task branch are saved; E. Detecting multi-task networks E1, Load the weights of the backbone feature extraction network This yields the Conv-Next-Net backbone feature extraction network with specific parameters. ); E2. Load the weights of the object detection branch network. This yields the target detection branch network Det( ​​with specific parameters). ); E3, Load the weights of the human keypoint detection branch network. This yields a human keypoint detection branch network K( with specific parameters). ); E4. Load the weights of the drivable region segmentation branch network. This yields the drivable region segmentation branch network SR( with specific parameters). ); E5. Input the image to be detected into the backbone feature extraction network to obtain the backbone extracted features as follows: Conv-Next-Net( )[input] input represents the image captured by the vehicle's onboard camera; This represents the obtained feature map; E6. Inputting the obtained feature maps into each task branch network simultaneously yields the following output: This indicates the final output; This represents a new task branch network with weights added; E7. The obtained output The corresponding target location, drivable area segmentation region, and human key points are marked in the initial camera input image to obtain the final detection map.

Images