Integrated decoder, multi-task network model and training method

Abstract

This invention discloses an integrated decoder, multi-task network model, and training method for in-vehicle vision systems. The integrated decoder comprises a topological structure composed of a cascade of two-level units, converting the multi-scale feature information output by a base network into a single-scale feature representation. The multi-task network model includes a base network, an integrated decoder, a target detection task head, and a scene segmentation task head. The method employs a balanced convergence method for the multi-task loss function, using empirical values of weight parameters to ensure balanced learning of multiple tasks during backpropagation. This invention improves the accuracy and stability of target detection and scene segmentation algorithms, while also enhancing their real-time performance without compromising their accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an integrated network model for target detection and semantic segmentation. Background Art

[0002] Currently, computer vision is a relatively low-cost area for advanced driver-assisted systems (ADAS). Its purpose is to use vision to obtain information about various objects in front of the vehicle and the position of lane lines, thereby providing the driver with collision and lane departure warnings, thereby protecting the driver's life. Given China's complex traffic environment, ADAS requires essential information, such as avoiding vehicles ahead and steering and stopping when changing lanes, and is an effective solution for reducing the incidence of accidents.

[0003] In recent years, deep learning-based object detection and scene segmentation algorithms have emerged, and all have high detection accuracy and real-time performance, such as EfficientDet, CenterNet, DeepLab, etc. The current integrated network design is completed by adding a segmentation task head based on DeepLab on the basis of the detection framework based on Yolo. The network structure is as follows: Figure 1 As shown in the figure, each task head has its own decoder. However, due to the complexity of the vehicle environment, including: overall image distortion caused by driving vibration, drastic changes in lighting, uneven scale of target detection frames, and incomplete lane lines due to wear, these problems will have some impact on the accuracy of existing target detection and scene segmentation algorithms.

[0004] Furthermore, for ADAS, the stability of the object detection algorithm's lower edge, the stability of lane line alternation during lane changes, and the real-time performance of the integrated algorithm are all crucial factors. The jitter stability of the lower edge of the detection frame affects the accuracy of vision-based ranging, which in turn affects the accuracy of ADAS forward collision warnings. The stability of lane line alternation during lane changes affects the accuracy of vision-based lane position deviation, which in turn affects ADAS lane deviation warnings. The real-time performance of the integrated algorithm directly affects the latency of the entire ADAS system's warnings. Summary of the Invention

[0005] In view of the analysis of the above problems, the present invention aims to provide an integrated decoder, multi-task network model and training method for vehicle-mounted vision systems to improve the accuracy, stability and real-time performance of target detection and scene segmentation algorithms.

[0006] The present invention is mainly achieved through the following technical solutions:

[0007] The integrated decoder provided by the present invention has a topological structure composed of a cascade of two-level units, which converts the multi-scale feature information output by the basic network into a single-scale feature information representation;

[0008] The secondary unit is composed of a sparse convolution layer, an upsampling deconvolution layer, an information fusion channel stacking layer and a secondary sparse convolution layer connected in series. The input of the secondary unit is a high-resolution feature map and a low-resolution feature map, and the output of the secondary unit is consistent with the size of the input high-resolution feature map.

[0009] Furthermore, the integrated decoder consists of a two-part network structure. The first part of the network structure includes three secondary units, which obtain fused feature maps downsampled to 16, 8, and 4 based on the 32, 16, and 8 downsampled feature maps output by the basic network. The second part of the network structure includes two secondary units, which obtain fused features downsampled to 4 based on the output of the first part of the network structure.

[0010] Furthermore, the first part of the network structure uses the 32 down-sampling feature maps and the 16 down-sampling feature maps output by the basic network to first enter the first secondary unit to output a 16 down-sampling fusion feature map, and then uses the 16 down-sampling fusion feature map and the 8 down-sampling feature map output by the basic network to enter the second secondary unit to output an 8 down-sampling fusion feature map, and uses the 8 down-sampling fusion feature map and the 4 down-sampling feature map output by the basic network to enter the third secondary unit to output a 4 down-sampling fusion feature map;

[0011] The second part of the network structure adopts the same cascade method for the two secondary units as the first part of the network structure, and downsamples the fusion feature map obtained by the first part of the network structure to 16, 8, and 4 to 4 fusion features.

[0012] As another aspect of the present invention, a multi-task network model based on the integrated decoder is provided, comprising a basic network, an integrated decoder, an object detection task head, and a scene segmentation task head;

[0013] The basic network selects a network model with a shallow depth and a large multi-scale cascade component, obtains the image of the current frame of the video stream, fixes the image to a specified resolution as the input of the model, and extracts multi-scale feature information;

[0014] The integrated decoder converts the multi-scale feature information output by the basic network into a single-scale feature information representation, that is, finally obtains a fusion feature with a downsampling of 4;

[0015] The target detection task head is the CenterNet target detection task head, and the scene segmentation task head is the FCN scene segmentation task head. The former is used to generate candidate boxes, and the latter generates scene masks to obtain the position information of the vehicle target and the lane line mask information.

[0016] The basic network selects the HRnetv2 model.

[0017] The target detection and scene segmentation task head outputs a feature map with a size of 10x40x104, which is downsampled to 4. The 10 layers of channels represent five variables, namely the detection box target center point category, detection box width, detection box height, detection box center point horizontal offset, and detection box center point vertical offset, and the segmentation mask is a background mask and a lane line mask for the left lane, left lane, right lane, and right lane, totaling five variables. In the two-dimensional point set with a resolution of 40x104, each point of the 10 channels is mapped back to the original image to obtain the target position and mask pixel category.

[0018] As another aspect of the present invention, a method for training the multi-task network model is provided, characterized in that it includes the following steps:

[0019] Step S1: Adjust the loss function and multi-task weight parameters in the training phase according to the integrated decoder structure;

[0020] Step S2: Use the CenterNet-based object detection loss function and the FCN-based scene segmentation loss function to converge the two tasks separately to obtain the multi-task loss function of the integrated network.

[0021] Step S3: Perform model training on the integrated network to obtain model parameters for balanced learning among subtasks.

[0022] The beneficial effects of the present invention are as follows:

[0023] The present invention proposes an integrated decoder based on multi-scale fusion, which unifies the scale problem of multi-task feature information.

[0024] This paper proposes an integrated network model for object detection and semantic segmentation suitable for in-vehicle vision system applications. Using high-resolution feature maps output by a multi-task integrated decoder, it improves the detection accuracy of small objects, occluded objects, and worn lane markings, while also enhancing algorithm precision. Using the integrated decoder's secondary unit to fuse the multi-scale features output by the base network can overcome issues such as drastic brightness changes and scene distortion in driving images.

[0025] The integrated decoder proposed in the present invention is shared by multiple task heads, which not only saves model space redundancy but also improves the real-time performance of model calculations, thereby providing early warning to drivers and helping them avoid driving risks.

[0026] The present invention adopts a balanced convergence method of multi-task loss function and uses empirical values of weight parameters to ensure the balance of the model for multi-task learning in back propagation.

[0027] The present invention improves the accuracy and stability of target detection and scene segmentation algorithms, and enhances the real-time performance of the algorithms without reducing the accuracy of the algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0029] Figure 1 This is the result diagram of the existing integrated network.

[0030] Figure 2 This is an integrated network structure diagram provided for an embodiment of the present invention.

[0031] Figure 3 This is a network structure diagram of the secondary unit of the integrated decoder provided in an embodiment of the present invention.

[0032] Figure 4 This is a display diagram of the output results of the multi-task network model of the present invention.

[0033] Figure 5 Output result display diagram for existing multi-task network model. DETAILED DESCRIPTION

[0034] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, which constitute a part of this application and are used to illustrate the principles of the present invention together with the embodiments of the present invention.

[0035] As an embodiment of the present invention, an integrated decoder is provided that converts multi-scale feature information output by a basic network into a single-scale feature information representation.

[0036] The integrated decoder is a topological structure composed of two-level cascade units, and the network structure is as follows: Figure 3 The secondary unit consists of a series of convolutional layers with a conv rate of 3, an upsampling deconvolution layer (deconv), a depthwise add layer for information fusion, and a secondary convolutional layer with a conv rate of 3. The inputs to the secondary unit are high-resolution and low-resolution feature maps, and the output of the secondary unit maintains the same size as the input high-resolution feature map.

[0037] Depend on Figure 2 It can be seen that the integrated decoder consists of two network structures. The first network structure includes three secondary units, which obtain fusion feature maps downsampled to 16, 8, and 4 based on the 32, 16, and 8 downsampled feature maps output by the basic network. The second network structure includes two secondary units, which obtain fusion features downsampled to 4 based on the output of the first network structure.

[0038] Specifically, the integrated decoder network structure is composed of the following:

[0039] The 32-downsampled feature map and the 16-downsampled feature map output by the base network are first fed into the first secondary unit to output a 16-downsampled fused feature map. The 16-downsampled fused feature map and the 8-downsampled feature map output by the base network are then fed into the second secondary unit to output an 8-downsampled fused feature map. The 8-downsampled fused feature map and the 4-downsampled feature map output by the base network are then fed into the third secondary unit to output a 4-downsampled fused feature map. In the second part, the fused feature maps with downsamples of 16, 8, and 4 are fed into two secondary units in the same cascaded manner, ultimately obtaining a fused feature with downsamples of 4.

[0040] Based on the above integrated decoder, an embodiment of the present invention provides a multi-task network model for target detection and semantic segmentation applied to an in-vehicle vision system, such as Figure 2 As shown, it includes the basic network, integrated decoder, target detection task head and scene segmentation task head.

[0041] The base network is a shallow network model with a large multi-scale cascade component. This embodiment uses the HRnetv2 model. The image of the current frame of the video stream is obtained and fixed to a specified resolution as input to the HRnetv2 model to extract multi-scale feature information.

[0042] The integrated decoder converts the multi-scale feature information output by the basic network into a single-scale feature information representation, that is, finally obtains a fusion feature with a downsampling of 4.

[0043] The target detection task head is the CenterNet target detection task head, and the scene segmentation task head is the FCN scene segmentation task head. The former is used to generate candidate boxes, and the latter generates scene masks to obtain the position information of the vehicle target and the lane line mask information.

[0044] Both the object detection and scene segmentation task heads output feature maps downsampled to 4, with a size of 10x40x104. These 10 channels represent five variables: the detection bounding box object center category, the detection bounding box width, the detection bounding box height, the horizontal offset of the detection bounding box center, and the vertical offset of the detection bounding box center. The segmentation mask also includes five variables: the background mask and the lane marking masks for the left lane, the left lane, the right lane, and the right lane. Each of these 10 channels represents a 2D point set with a resolution of 40x104. When each point is mapped back to the original image, the physical meaning of the object location and mask pixel category is obtained.

[0045] Based on the above-mentioned multi-task network model for object detection and semantic segmentation, a learning and training method is provided as an embodiment of the present invention. It adopts a balanced convergence method for multi-task loss functions and uses empirical values of weight parameters to ensure that the model learns balanced multi-tasks during backpropagation. It includes the following steps:

[0046] Step S1: Adjust the loss function and multi-task weight parameters in the training phase based on the integrated decoder structure. The main parameters involved are:

[0047] (1) Target detection loss function: L hm , L wh , L reg

[0048] (2) Scene segmentation loss function: L seg

[0049] (3) Integrated network loss function: L

[0050] (4) Category prediction and true value of the target detection box center position: M xy

[0051] (5) Predicted and true values of the width and height of the target detection box: w i , h i

[0052] (6) The predicted value and true value of the displacement between the center position of the target detection frame and the feature point position: reg_x i

[0053] (7) Predicted value and true value of mask pixel category: y i

[0054] (8) Loss function weight parameter: λ hm ,λ wh ,λ reg ,λ seg

[0055] Step S2: Use the CenterNet-based object detection loss function and the FCN-based scene segmentation loss function to converge the two tasks separately, and obtain the multi-task loss function of the integrated network. The specific formula is as follows:

[0056]

[0057]

[0058]

[0059]

[0060] L=λ hm L hm +λ wh L wh +λ reg L reg +λ seg L seg (5)

[0061] Step S3: Perform model training on the integrated network according to the above steps to obtain model parameters for balanced learning among subtasks.

[0062] The result graphs of the multi-task network model proposed in this invention and the output of the current multi-task network model are compared. Figure 4 and Figure 5 As shown, in the scene shown in the figure, the two vehicles in the opposite lane are blocked by the isolation belt. The present invention can detect the two blocked vehicles, while the current integrated model can only detect one vehicle. Among the vehicles traveling in the same direction, there is an SUV in the distance in front of the screen. Compared with the current model, the detection result of the model proposed by the present invention will be more stable and the jitter level will be smaller at the bottom edge of the detection frame. Because the current vehicle is in the moment of changing lanes, the current algorithm is not accurate enough for the lane line segmentation. It can be seen from the figure that the segmentation of the adjacent left lane is incomplete and the left lane mask is incorrectly classified. Although the classification of the right lane is correct, the segmentation mask is discontinuous and the adjacent right lane is not segmented. From the comparison, it can be seen that the lane line segmentation result proposed by the present invention is not only correct in lane line classification, but also continuous and complete in segmentation mask.

[0063] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0064] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A multi-task network model of an integrated decoder, characterized in that: The multi-task network model includes a base network, an integrated decoder, an object detection task head, and a scene segmentation task head; the integrated decoder is a topological structure composed of a cascade of two-level units, which converts the multi-scale feature information output by the base network into a single-scale feature information representation; The secondary unit is composed of a sparse convolution layer, an upsampling deconvolution layer, an information fusion channel stacking layer and a secondary sparse convolution layer in series. The input of the secondary unit is a high-resolution feature map and a low-resolution feature map. The output of the secondary unit is consistent with the size of the input high-resolution feature map. The integrated decoder consists of a two-part network structure. The first part of the network structure includes three secondary units. Based on the 32, 16, and 8 downsampling feature maps output by the basic network, fusion feature maps with downsampling of 16, 8, and 4 are obtained. The second part of the network structure includes two secondary units. Based on the output of the first part of the network structure, a fusion feature with downsampling of 4 is obtained. The first part of the network structure uses the 32 downsampling feature map and the 16 downsampling feature map output by the basic network to first enter the first secondary unit to output a 16 downsampling fusion feature map, and then uses the 16 downsampling fusion feature map and the 8 downsampling feature map output by the basic network to enter the second secondary unit to output an 8 downsampling fusion feature map, and uses the 8 downsampling fusion feature map and the 4 downsampling feature map output by the basic network to enter the third secondary unit to output a 4 downsampling fusion feature map. The second part of the network structure adopts the same cascade method for the two secondary units as the first part of the network structure, and downsamples the fusion feature map obtained by the first part of the network structure to 16, 8, and 4 to 4 fusion features; The basic network selects a network model with a shallow depth and a large multi-scale cascade component, obtains the image of the current frame of the video stream, fixes the image to a specified resolution as the input of the model, and extracts multi-scale feature information; The integrated decoder converts the multi-scale feature information output by the basic network into a single-scale feature information representation, that is, finally obtains a fusion feature with a downsampling of 4; The target detection task head is the CenterNet target detection task head, and the scene segmentation task head is the FCN scene segmentation task head. The former is used to generate candidate boxes, and the latter generates scene masks to obtain the position information of the vehicle target and the lane line mask information.

2. The multi-task network model according to claim 1, wherein: The basic network selects the HRnetv2 model.

3. The multi-task network model according to claim 1 or 2, characterized in that: The target detection and scene segmentation task head outputs a feature map with a size of 10x40x104, which is downsampled to 4. The 10 layers of channels represent five variables, namely the detection box target center point category, detection box width, detection box height, detection box center point horizontal offset, and detection box center point vertical offset, and the segmentation mask is a background mask and a lane line mask for the left lane, left lane, right lane, and right lane, totaling five variables. In the two-dimensional point set with a resolution of 40x104, each point of the 10 channels is mapped back to the original image to obtain the target position and mask pixel category.

4. The training method of the multi-task network model according to claim 1 is characterized in that: The following steps are involved: Step S1: According to the integrated decoder structure, adjust the loss function and multi-task weight parameters in the training phase, involving the following parameters: (1) Target detection loss function: L hm , L wh , L reg (2) Scene segmentation loss function: L seg (3) Integrated network loss function: L (4) Category prediction and true value of the target detection box center position: M xy (5) Predicted and true values of the width and height of the target detection box: w i , h i (6) The predicted value and true value of the displacement between the center position of the target detection frame and the feature point position: reg_x i (7) Predicted value and true value of mask pixel category: y i (8) Loss function weight parameter: λ hm ,λ wh ,λ reg ,λ seg Step S2: Use the CenterNet-based object detection loss function and the FCN-based scene segmentation loss function to converge the two tasks separately, and obtain the multi-task loss function of the integrated network. The specific formula is as follows: L=λ hm L hm +λ wh L wh +λ reg L reg +λ seg L seg (5) Step S3: Perform model training on the integrated network to obtain model parameters for balanced learning among subtasks.

Images