Multi-task joint image recognition method

Abstract

The invention relates to a multi-task joint image recognition method, and relates to the technical field of computer vision and deep learning. Comprises: constructing a training sample; constructing a multi-task joint image recognition model, setting model training hyper-parameters, inputting training data into the feature extraction backbone network, and obtaining a backbone network output feature map; inputting the feature map into a check network, and outputting a multi-scale feature map; inputting the multi-scale feature map into a multi-task detection branch to obtain a plurality of task prediction results; calculating and evaluating a loss value according to the result and a data set label, and iteratively training and optimizing through an optimizer; obtaining a model weight and then loading test data evaluation; a target detection result is post-processed by adopting non-maximum suppression, a drivable region segmentation result is subjected to discontinuous and small hole filling by adopting a morphological operation method to extract boundary points, a lane line segmentation result is subjected to clustering and sliding window search, and lane line fitting is performed after isolated feature point clusters are removed; and finally, reasoning the model deployment controller.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving environment perception, and specifically to a multi-task joint image recognition method. Background Art

[0002] In the operation scenario of an autonomous vehicle, it is necessary to simultaneously detect vehicles, pedestrians, traffic lights, traffic signs, obstacle information, and identify lane lines and drivable areas on the road. In the existing technology, multiple models are used to solve the needs of multiple types of detection tasks in the same application scenario. However, each existing detection task is executed independently, which wastes a certain amount of memory bandwidth and cannot utilize the relevant semantic features between different detection tasks, resulting in high hardware resource consumption and low recognition accuracy during model reasoning. Summary of the invention

[0003] In order to solve the problems existing in the background technology, the present invention aims to provide a multi-task joint image recognition method, which can simultaneously perform lane line recognition, drivable area and road target detection.

[0004] Solutions for solving problems

[0005] A multi-task joint image recognition method, the method comprising:

[0006] Obtain and annotate the returned image data to build a training data set;

[0007] Constructing a multi-task joint image recognition model, wherein the deep network model of the multi-task joint image recognition model includes a feature extraction backbone network, a Neck feature fusion network and a multi-task joint recognition network detection branch connected in sequence;

[0008] Load training data, set hyperparameters, optimize model training and save corresponding weights, where the hyperparameters include learning rate, optimizer, number of iterations, IoU threshold and anchor threshold;

[0009] Load the model weights and evaluate the accuracy. The evaluation accuracy indicators for the segmentation task are pixel accuracy and average intersection-over-union ratio, and for the target detection task is the average accuracy under an intersection-over-union ratio of 0.5;

[0010] The output result is post-processed, and the post-processing includes target detection task post-processing, drivable area segmentation task post-processing and lane line detection segmentation task post-processing.

[0011] Furthermore, the construction of the training data set specifically includes: acquiring image information of a vehicle running road scene, synchronously annotating target detection information and road structure semantic segmentation information in the image, and obtaining a training data set; the target detection information includes the 2D box coordinates of the object to be annotated in the image and the category of the object to be annotated, and the road structure semantic segmentation information includes the pixel points where the lane lines and the drivable area are located.

[0012] Furthermore, the shared feature extraction backbone network takes the image data as input and processes to obtain global feature information and local feature information of different scales for subsequent detection task processing;

[0013] The Neck feature fusion network processes the features extracted by the backbone network and outputs them to the multi-task joint recognition network detection branch for specific task learning;

[0014] The multi-task joint recognition network detection branch includes decoders for three tasks: target detection, drivable area segmentation, and lane line segmentation. The target detection task decoder uses a multi-scale detection technology based on anchor frames. The path aggregation PANet layer transmits image features from bottom to top, and the multi-scale fusion feature map obtained is used for detection. The drivable area segmentation and lane line segmentation tasks use the same network structure, and the output feature size is the input image size through three upsampling processes.

[0015] Furthermore, the feature extraction backbone network uses CSPDarknet, the Neck layer uses the spatial pyramid SPP, and the SPP model is used to generate and fuse features of different scales.

[0016] Furthermore, the model training optimization includes: dividing the training set and the test set in an 8:2 ratio based on the constructed data set; for the labels of the target detection task in the training data set, using the k-means clustering method to obtain the anchor settings that match the detection box of the training data; loading the training data set, initializing the hyperparameters, the hyperparameters including the learning rate, optimizer, number of iterations, IOU threshold, and anchor threshold; iteratively optimizing the model, using image flipping, HSV color gamut conversion, and Mosaic method to augment the image, using the augmented image data as input, calculating the model prediction result and the label error value, the error value is the weighted sum of the target detection loss and the loss of the drivable area and lane line segmentation tasks;

[0017] The overall loss calculation formula is:

[0018] L all =γ1L det +γ2L da-seg +γ3L ll-seg

[0019] The target detection loss is the weighted sum of classification loss, confidence loss, and position loss, and the calculation formula is:

[0020]

[0021] The drivable area loss uses cross entropy loss, and the calculation formula is:

[0022] L da-seg =L ce

[0023] The lane line loss uses cross entropy loss and intersection-over-union loss, and the calculation formula is:

[0024]

[0025] L ll-seg =L ce +L IoU

[0026] ADAM is used to optimize the loss function error during model training.

[0027] Furthermore, the output results are post-processed, specifically including: target detection results, drivable area segmentation mask map and lane line segmentation mask map; the target detection results are post-processed using the non-maximum suppression method, and the output results include target object category, confidence and detection box coordinate points; the output results of the drivable area and lane line segmentation tasks are further post-processed, and the drivable area segmentation mask map is binarized and then filled with discontinuities and small holes using closed operations; the lane line segmentation task detection results are binarized and then clustered and searched using DBSCAN clustering and sliding windows to eliminate isolated feature point clusters to obtain several lane line point sets.

[0028] Furthermore, the method also includes: model deployment reasoning, pruning and compressing the model weight file after the iterative optimization is completed, and performing int8 quantization, and deploying the model to the edge controller for reasoning.

[0029] Beneficial effects:

[0030] The beneficial effects of the above technical solution are:

[0031] The multi-task joint image recognition detection model can simultaneously perform lane line, road structure recognition of drivable areas, and road target detection of vehicles, pedestrians, traffic lights, traffic signs, etc. It can realize the synchronous recognition of multiple categories in the autonomous driving perception task, and can obtain all target information in the image to be tested at one time, saving hardware resources and video memory resources. In addition, the invention has diversified detection scales, strong robustness, high detection accuracy and fast speed in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1is an application schematic diagram provided by the first embodiment of the present invention;

[0033] Figure 2 It is a schematic diagram of the model structure provided by the first embodiment of the present invention. DETAILED DESCRIPTION

[0034] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0035] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0036] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0037] Embodiment 1:

[0038] Figure 1 This is a flowchart of a multi-task joint image recognition detection method provided by the first embodiment of the present invention. The method of this embodiment can be implemented by software and can be integrated into a controller. The method of this embodiment specifically includes:

[0039] Step S10: Training data construction, deploying the actual operating environment according to the algorithm, placing camera sensors on the collection vehicle, implementing camera topic recording based on ROS, sorting the returned data and annotating the road structure and target information. The road structure is annotated with closed curves, and the target information is annotated with 2D box selection.

[0040] Step S20: Model construction, such as Figure 2 The structure of the multi-task joint image recognition model shown in the figure includes a feature extraction backbone network, a Neck feature fusion network, and a multi-task joint recognition network detection branch connected in sequence. The feature extraction backbone network uses CSPDarknet, the Neck layer uses a spatial pyramid (SPP), and the SPP model is used to generate and fuse features of different scales.

[0041] The detection head includes decoders for three tasks: target detection, drivable area segmentation, and lane line segmentation. The target detection task decoder uses multi-scale detection technology based on anchor frames. The path aggregation PANet layer transmits image features from bottom to top, and the multi-scale fusion feature map obtained is used for detection.

[0042] The drivable area segmentation and lane line segmentation tasks use the same network structure, and the output feature size is the same as the input image size through three upsampling processes.

[0043] Step S30: Model training optimization, based on the construction of the data set, the training set and the test set are divided into 8:2 ratios. For the labels of the target detection task in the training data set, the k-means clustering method is used to obtain the anchor settings that match the detection box of the training data. Load the training data set and initialize the hyperparameters, which include the learning rate, optimizer, number of iterations, IOU threshold, and anchor threshold. Iterate and optimize the model, use image flipping, HSV color gamut transformation, and Mosaic method to augment the image, use the augmented image data as input, calculate the model prediction result and the label error value, and the error value is the weighted sum of the target detection loss and the loss of the drivable area and lane line segmentation tasks.

[0044] The overall loss calculation formula is:

[0045] L all =γ1L det +γ2L da-seg +γ3L ll-seg

[0046] The target detection loss is the weighted sum of classification loss, confidence loss, and position loss, and the calculation formula is:

[0047]

[0048] The drivable area loss uses cross entropy loss, and the calculation formula is:

[0049] L da-seg =L ce

[0050] The lane line loss uses cross entropy loss and intersection-over-union loss, and the calculation formula is:

[0051]

[0052] L ll-seg =L ce +L IoU

[0053] ADAM is used to optimize the loss function error during model training.

[0054] Step S40: Model accuracy evaluation, loading model weights and reading test data sets, evaluating model recognition accuracy, drivable area and lane line segmentation task evaluation indicators include pixel accuracy and average intersection-over-union ratio, and target detection task indicators include average accuracy under an intersection-over-union ratio of 0.5.

[0055] Step S50: Output result post-processing, model prediction output includes target detection results, drivable area segmentation mask map, lane line segmentation mask map. The target detection result is post-processed using the non-maximum suppression method, and the output results include target category, confidence, and detection frame coordinate points. The output results of the drivable area and lane line segmentation tasks are further post-processed, and the drivable area segmentation mask map is binarized and closed operations are used to fill discontinuities and small holes. After the lane line segmentation task detection results are binarized, cluster search is performed using DBSCAN clustering and sliding windows to eliminate isolated feature point clusters and obtain several lane line point sets.

[0056] Step S60: Model deployment reasoning, pruning and compressing the model weight file after iterative optimization, and performing int8 quantization, and deploying the model to the edge controller for reasoning;

[0057] The multi-task joint image detection model is deployed to the vehicle embedded platform, collected by camera, and the model is run in the embedded platform to detect the lane lines in front of the car, the road structure of the drivable area, and the information of vehicles, pedestrians, traffic lights, and traffic signs.

[0058] Embodiment 2:

[0059] The difference between this embodiment and the multi-task joint image recognition method described in the first embodiment is that a recording device with acquisition and storage functions is used in the construction of training data.

[0060] It should be noted that in the multi-task joint image recognition and detection method provided in an embodiment of the present invention, the specific implementation of each of the above steps and the technical effects produced are the same as those in the aforementioned method embodiment. For the sake of brief introduction, the parts not mentioned in this embodiment can be referred to the corresponding contents in the aforementioned method embodiment.

[0061] Embodiment 3:

[0062] The difference between this embodiment and the multi-task joint image recognition detection method described in the first embodiment is that the feature extraction backbone network uses EfficientNet.

[0063] It should be noted that in the multi-task joint image recognition and detection method provided in an embodiment of the present invention, the specific implementation of each of the above steps and the technical effects produced are the same as those in the aforementioned method embodiment. For the sake of brief introduction, the parts not mentioned in this embodiment can be referred to the corresponding contents in the aforementioned method embodiment.

[0064] Embodiment 4:

[0065] The difference between this embodiment and the multi-task joint image recognition method described in the first embodiment is that the feature extraction backbone network uses a transformer encoder.

[0066] It should be noted that in the multi-task joint image recognition and detection method provided in an embodiment of the present invention, the specific implementation of each of the above steps and the technical effects produced are the same as those in the aforementioned method embodiment. For the sake of brief introduction, the parts not mentioned in this embodiment can be referred to the corresponding contents in the aforementioned method embodiment.

[0067] Embodiment 5:

[0068] The difference between this embodiment and the multi-task joint image recognition method described in the first embodiment is that the model training adopts k-fold cross validation, where k is 10.

[0069] It should be noted that in the multi-task joint image recognition and detection method provided in an embodiment of the present invention, the specific implementation of each of the above steps and the technical effects produced are the same as those in the aforementioned method embodiment. For the sake of brief introduction, the parts not mentioned in this embodiment can be referred to the corresponding contents in the aforementioned method embodiment.

[0070] Embodiment 6:

[0071] The difference between this embodiment and the multi-task joint image recognition detection method described in the first embodiment is that the target detection head uses anchorfree

[0072] It should be noted that in the multi-task joint image recognition and detection method provided in an embodiment of the present invention, the specific implementation of each of the above steps and the technical effects produced are the same as those in the aforementioned method embodiment. For the sake of brief introduction, the parts not mentioned in this embodiment can be referred to the corresponding contents in the aforementioned method embodiment.

[0073] Embodiment 7:

[0074] The difference between this embodiment and the multi-task joint image recognition detection method described in the first embodiment is that in the model construction process, the neck layer uses SPP+PANet.

[0075] It should be noted that in the multi-task joint image recognition and detection method provided in an embodiment of the present invention, the specific implementation of each of the above steps and the technical effects produced are the same as those in the aforementioned method embodiment. For the sake of brief introduction, the parts not mentioned in this embodiment can be referred to the corresponding contents in the aforementioned method embodiment.

[0076] Embodiment 8:

[0077] The difference between this embodiment and the multi-task joint image recognition detection method described in the first embodiment is that the optimizer uses SGD during the model optimization process.

[0078] It should be noted that in the multi-task joint image recognition and detection method provided in an embodiment of the present invention, the specific implementation of each of the above steps and the technical effects produced are the same as those in the aforementioned method embodiment. For the sake of brief introduction, the parts not mentioned in this embodiment can be referred to the corresponding contents in the aforementioned method embodiment.

[0079] Embodiment 9:

[0080] The difference between this embodiment and the multi-task joint image recognition detection method described in the first embodiment is that the clustering method uses k-means in the post-processing process of the lane segmentation results.

[0081] It should be noted that in the multi-task joint image recognition and detection method provided in an embodiment of the present invention, the specific implementation of each of the above steps and the technical effects produced are the same as those in the aforementioned method embodiment. For the sake of brief introduction, the parts not mentioned in this embodiment can be referred to the corresponding contents in the aforementioned method embodiment.

[0082] Embodiment 10:

[0083] The difference between this embodiment and the multi-task joint image recognition detection method described in the first embodiment is that the model is deployed in the cloud to realize the prediction function, the original data is uploaded to the cloud, and the inference results are sent to the vehicle.

[0084] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0085] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

Claims

1. A multi-task joint image recognition method, characterized in that: The method comprises: Obtain and annotate the returned image data to build a training data set; Constructing a multi-task joint image recognition model, wherein the deep network model of the multi-task joint image recognition model includes a feature extraction backbone network, a Neck feature fusion network and a multi-task joint recognition network detection branch connected in sequence; Load training data, set hyperparameters, optimize model training and save corresponding weights, where the hyperparameters include learning rate, optimizer, number of iterations, IoU threshold and anchor threshold; Load the model weights and evaluate the accuracy. The evaluation accuracy indicators for the segmentation task are pixel accuracy and average intersection-over-union ratio, and for the target detection task is the average accuracy under an intersection-over-union ratio of 0.5; The output result is post-processed, and the post-processing includes target detection task post-processing, drivable area segmentation task post-processing and lane line detection segmentation task post-processing.

2. A multi-task joint image recognition method according to claim 1, characterized in that: The construction of the training data set specifically includes: acquiring image information of a vehicle running road scene, synchronously annotating target detection information and road structure semantic segmentation information in the image, and obtaining a training data set; the target detection information includes the 2D box coordinates of the object to be annotated in the image and the category of the object to be annotated, and the road structure semantic segmentation information includes the pixel points where the lane line and the drivable area are located.

3. The multi-task joint image recognition method according to claim 1, characterized in that: The shared feature extraction backbone network takes image data as input and processes to obtain global feature information and local feature information of different scales for subsequent detection task processing; The Neck feature fusion network processes the features extracted by the backbone network and outputs them to the multi-task joint recognition network detection branch for specific task learning; The multi-task joint recognition network detection branch includes decoders for three tasks: target detection, drivable area segmentation, and lane line segmentation. The target detection task decoder uses a multi-scale detection technology based on anchor frames. The path aggregation PANet layer transmits image features from bottom to top, and the multi-scale fusion feature map obtained is used for detection. The drivable area segmentation and lane line segmentation tasks use the same network structure, and the output feature size is the input image size through three upsampling processes.

4. The multi-task joint image recognition method according to claim 1, characterized in that: The feature extraction backbone network uses CSPDarknet, the Neck layer uses the spatial pyramid SPP, and the SPP model is used to generate and fuse features of different scales.

5. The multi-task joint image recognition method according to claim 1, characterized in that: The model training optimization includes: dividing the training set and the test set in an 8:2 ratio based on the constructed data set; for the labels of the target detection task in the training data set, using the k-means clustering method to obtain the anchor settings that match the detection frame of the training data; loading the training data set, initializing the hyperparameters, the hyperparameters including the learning rate, optimizer, number of iterations, IOU threshold, and anchor threshold; iteratively optimizing the model, using image flipping, HSV color gamut conversion, and Mosaic method to augment the image, using the augmented image data as input, calculating the model prediction result and the label error value, the error value is the weighted sum of the target detection loss and the loss of the drivable area and lane line segmentation task; The overall loss calculation formula is: L all =γ1L det +γ2L da-seg +γ3L ll-seg The target detection loss is the weighted sum of classification loss, confidence loss, and position loss, and the calculation formula is: The drivable area loss uses cross entropy loss, and the calculation formula is: L da-seg =L ce The lane line loss uses cross entropy loss and intersection-over-union loss, and the calculation formula is: L ll-seg =L ce +L IoU ADAM is used to optimize the loss function error during model training.

6. A multi-task joint image recognition method according to claim 1, characterized in that: The output result post-processing specifically includes: target detection results, drivable area segmentation mask map and lane line segmentation mask map; the target detection results are post-processed using the non-maximum suppression method, and the output results include target object category, confidence and detection frame coordinate points; the output results of the drivable area and lane line segmentation tasks are further post-processed, and the drivable area segmentation mask map is binarized and then filled with discontinuities and small holes using closed operations; the lane line segmentation task detection results are binarized and then clustered and searched using DBSCAN clustering and sliding windows to eliminate isolated feature point clusters and obtain a number of lane line point sets.

7. The multi-task joint image recognition method according to claim 1, characterized in that: The method also includes: model deployment reasoning, pruning and compressing the model weight file after the iterative optimization is completed, performing int8 quantization, and deploying the model to the edge controller for reasoning.

Images