Driving assistance system and driving assistance calculation method
By combining object detection and semantic segmentation into a deep learning model based on block-based image detection, the problem of long inference time and cumbersome post-processing in semantic segmentation models is solved, resulting in faster detection speed and lower resource consumption.
Patent Information
- Application Number
- CN202410038311.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2024-01-10
- Publication Date
- 2024-11-15
AI Technical Summary
In advanced driver assistance systems, the semantic segmentation model has a long inference time and complicated post-processing steps, resulting in an excessively long overall detection time.
A deep learning model for block-based image detection is adopted, omitting the encoding and decoding architecture. It directly detects lane lines and drivable areas through block line segment parameters, and merges object detection and semantic segmentation tasks into a single model.
It reduced post-processing time consumption, lowered model complexity and memory requirements, and improved inference speed and processing efficiency. The accuracy decreased by only 0.1%, but the inference time was reduced by 47.9%.
Smart Images

Figure CN118953365A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a driving assistance system and a driving assistance calculation method, and in particular to a driving assistance system and a driving assistance calculation method based on a deep neural network architecture. Background Art
[0002] Advanced Driver Assistance Systems (ADAS) usually perform the tasks of object detection and image segmentation. Object detection includes pedestrians, traffic lights, traffic signs, two-wheelers and four-wheelers, and image segmentation includes lane lines and drivable areas. Advanced Driver Assistance Systems (ADAS) usually perform the tasks of object detection and image segmentation. Object detection includes pedestrians, traffic lights, traffic signs, two-wheelers and four-wheelers, and image segmentation includes lane lines and drivable areas.
[0003] However, most advanced driver assistance systems require the above-mentioned detection and recognition functions. Therefore, they require two deep learning models with different architectures to implement. In addition to requiring twice the inference time, the complexity of the semantic segmentation model is higher than that of the object detection deep learning model.
[0004] For example, the detection task of the driving area is mainly carried out by the semantic segmentation model. Most semantic segmentation models are usually designed to go through the encoding and decoding process. The result after the model inference is to distinguish the categories in the form of pixels. It must be grouped and fitted with polynomials through post-processing to obtain lane lines and drivable areas.
[0005] Based on the above, it can be seen that the current semantic segmentation model not only takes a huge amount of time during model inference, but also takes extra time for post-processing steps. If the model encoding, decoding and tedious post-processing steps can be omitted, the time of the entire detection task will be saved. Therefore, if the semantic segmentation task can be implemented by integrating the object detection deep learning model architecture, the complexity of the original semantic segmentation process will be simplified, and the goal of achieving dual functions can be achieved with one inference. Summary of the invention
[0006] The present invention provides a multi-task detection and recognition technology based on a single deep learning model, including object detection and recognition and semantic segmentation. The model adopts a block-based image detection and recognition method, omitting the encoding and decoding architecture. The model will perform line segment detection on multiple blocks in the image, and directly provide the polynomial coefficients, categories, and left and right upper and lower limits of the block line segments. In this way, the post-processing stage only needs to analyze the relationship between each block and the adjacent block line segments to obtain the lane lines and drivable areas, thereby effectively reducing the time consumption of post-processing.
[0007] The present invention merges semantic segmentation and object detection directly into a single target platform. In terms of all functions, the architecture merger provides fewer layers, which means fewer parameters and less memory requirements. The inference time of the architecture merger also provides faster time cost and fewer processing operations.
[0008] The present invention proposes a method suitable for embedded systems, which can detect lane lines, drivable areas, etc. and various objects (two-wheeled vehicles, four-wheeled vehicles, pedestrians, traffic lights, etc.) on images in real time. Lane lines and drivable areas are detected in real time through block line segments, and the polynomials, categories, and left and right upper and lower limits of each block line segment are extracted. The present invention can save the time of grouping lane line pixels and then fitting polynomials after semantic segmentation, and will not affect the results of an entire line due to the confidence level of end-to-end polynomial lane line detection.
[0009] Specifically, the present invention provides a driving assistance system, characterized in that the system utilizes a deep neural network architecture, which is used to perform object detection and semantic segmentation detection and recognition. The system includes: an image capture module, configured to capture an image; an image cutting module, configured to divide the image into multiple block images; a processing module, configured to execute the deep neural network architecture based on the block images to construct multiple common feature maps for object detection and semantic segmentation, multiple object frames and multiple block line segment parameters, and configured to perform filtering and merging based on the block line segment parameters; an output module, configured to output information on object detection and semantic segmentation detection.
[0010] In one embodiment, the image capture module of the driving assistance system includes at least one RGB camera.
[0011] In one embodiment, the processing module of the above-mentioned driving assistance system includes an object detection module and a line segment detection module, the object detection module is configured to execute an evaluation method for object detection, the evaluation method for object detection includes using the intersection over union (IoU) method for evaluation, and the line segment detection module is configured to execute an evaluation method for line segment detection, and the evaluation method for line segment detection includes using Tusimple's indicators for evaluation. In one embodiment, the processing module of the above-mentioned driving assistance system includes an object detection module and a line segment detection module, the object detection module is configured to execute an evaluation method for object detection, the evaluation method for object detection includes using the intersection over union (IoU) method for evaluation, and the line segment detection module is configured to execute an evaluation method for line segment detection, and the evaluation method for line segment detection includes using Tusimple's indicators for evaluation.
[0012] In one embodiment, the deep neural network architecture of the driving assistance system includes implementing an algorithm, which includes a loss function for line segment detection, a loss function used in target detection, and a loss function for object box regression.
[0013] The present invention further provides a driving assistance operation method, characterized in that the method utilizes a deep neural network architecture, which is used to perform object detection and semantic segmentation detection and recognition. The method includes the following steps: Step S1: Capturing an image; Step S2: Dividing the image into multiple block images; Step S3: Executing the deep neural network architecture based on the block images to construct multiple common feature maps for object detection and semantic segmentation, multiple object frames and multiple block line segment parameters; Step S4: Executing filtering steps and merging based on the block line segment parameters; Step S5: Outputting object detection and semantic segmentation information.
[0014] In one embodiment, step S1 of the above method is to obtain the image through at least one RGB camera.
[0015] In one embodiment, step S3 of the above method includes executing an object detection evaluation method and a line segment detection evaluation method, wherein the object detection evaluation method includes using an intersection over union (IoU) method for evaluation, and the line segment detection evaluation method includes using a Tusimple indicator for evaluation.
[0016] In one embodiment, step S3 of the above method includes implementing an algorithm, wherein the algorithm includes a loss function for line segment detection, a loss function used in target detection, and a loss function for object box regression.
[0017] The deep neural network architecture used in the present invention focuses on the design of the architecture and loss function, reducing the Backbone and Neck time of the task detection model, which is suitable for real-time processing of embedded systems, and divides the picture into multiple blocks for lane line and drivable area detection, and maintains a multi-task detection model with a high correct detection rate.
[0018] The present invention is based on an object detection framework and fragment-based line segment detection. By constructing multiple common feature maps for object detection and semantic segmentation, multiple object frames and multiple block line segment parameters, the same model can be used to perform object detection and semantic segmentation in one inference.
[0019] In order to make the above and other purposes, features and advantages of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 It is a schematic diagram of a driving assistance system according to an embodiment of the present invention.
[0021] Figure 2 It is a flow chart of a driving assistance calculation method according to an embodiment of the present invention.
[0022] Figure 3 It is a schematic diagram of the fusion of object detection and semantic segmentation architecture according to an embodiment of the present invention.
[0023] Figure 4 It is a schematic diagram of block-based line segment detection according to an embodiment of the present invention.
[0024] Figure 5 Schematic diagram of an embodiment of the present invention in which the loss function of object box regression adopts CIOU Loss (Complete IOU Loss).
[0025] Figure 6 FIG. 1 is a schematic diagram of an object detection evaluation method according to an embodiment of the present invention, using Intersection Over Union (IoU).
[0026] Figure 7 It is a schematic diagram of a line segment detection result according to an embodiment of the present invention.
[0027] Figure 8 This is a test verification result of an embodiment of the present invention.
[0028] 1: Driving assistance system
[0029] 10: Image capture module
[0030] 11: Image cutting module
[0031] 12: Processing module
[0032] 121: Object detection module
[0033] 122: Line segment detection module
[0034] 13: Output module
[0035] S1-S4: Steps DETAILED DESCRIPTION
[0036] In order to understand the technical features, contents and advantages of the present invention and the effects that can be achieved, this creation is hereby described in detail in the form of an embodiment in conjunction with the accompanying drawings. The figures used therein are only for the purpose of illustration and auxiliary description, and may not be the actual proportion and precise configuration after the implementation of the present invention. Therefore, the proportion and configuration relationship of the attached figures should not be interpreted or limited to the scope of rights of the present invention in actual implementation. In order to understand the technical features, contents and advantages of the present invention and the effects that can be achieved, this creation is hereby described in detail in the form of an embodiment in conjunction with the accompanying drawings. The figures used therein are only for the purpose of illustration and auxiliary description, and may not be the actual proportion and precise configuration after the implementation of the present invention. Therefore, the proportion and configuration relationship of the attached figures should not be interpreted or limited to the scope of rights of the present invention in actual implementation.
[0037] Embodiment 1: Driving assistance system
[0038] See also Figure 1 , Figure 1 The present invention provides a driving assistance system 1, which uses a deep neural network architecture to perform object detection and semantic segmentation detection and recognition. The system includes: an image capture module 10, configured to capture an image; an image segmentation module 11, configured to segment the image into a plurality of block images; a processing module 12, configured to execute the deep neural network architecture according to the block images to construct a plurality of common feature maps for object detection and semantic segmentation, a plurality of object frames and a plurality of block line segment parameters, and configured to perform filtering and merging according to the block line segment parameters; an output module 13, configured to output information of object detection and semantic segmentation detection.
[0039] The processing module 12 of the driving assistance system 1 includes an object detection module 121 and a line segment detection module 122. The object detection module 121 is configured to execute an object detection evaluation method, and the object detection evaluation method includes using the Intersection Over Union (IoU) method for evaluation. The line segment detection module 122 is configured to execute a line segment detection evaluation method, and the line segment detection evaluation method includes using Tusimple indicators for evaluation.
[0040] Embodiment 2: Driving assistance calculation method
[0041] See also Figure 2 , Figure 2 The flowchart of the driving assistance computing method of one embodiment of the present invention is provided. The present invention provides a driving assistance computing method, which utilizes a deep neural network architecture to realize object detection and semantic segmentation detection and recognition, and the method comprises the following steps: Step S1: capturing an image; Step S2: dividing the image into a plurality of block images; Step S3: executing the deep neural network architecture according to the block images to construct a plurality of common feature maps for object detection and semantic segmentation, a plurality of object frames and a plurality of block line segment parameters; Step S4: executing filtering steps and merging according to the block line segment parameters; Step S5: outputting the information of object detection and semantic segmentation.
[0042] Example 3: Fusion of object detection and semantic segmentation architecture
[0043] This embodiment proposes a deep neural network architecture that combines object detection and semantic segmentation architecture, such as Figure 3 As shown, the architecture can achieve two different types of detection and recognition: object detection and semantic segmentation through a single deep neural network architecture.
[0044] This embodiment is based on the object detection architecture and is extended to achieve the semantic segmentation function. This detection method is different from the previous semantic segmentation. Semantic segmentation is a pixel-level prediction, while this embodiment detects the boundary of semantic segmentation, and its boundary is composed of line segment detection through the fragment, thereby realizing the semantic segmentation function.
[0045] Line segment detection is the line segment parameter of the block, which can be applied to lane line detection or boundary detection of the drivable area. In this embodiment, the deep neural network architecture performs two tasks, namely four-wheel vehicle detection and identification and lane line detection and identification. The deep neural network architecture is as follows Figure 3 As shown, the architecture used is a pyramid network with FPN (Feature Pyramid Network) plus PAN (Pixel Aggregation Network), which can enable the deep and shallow information in the architecture to communicate. Therefore, in order to make up for the problem of being unable to locate in the classification architecture, the output of the additional network architecture (detection head) contains information on objects and line segments (semantic segmentation). The object categories that can also be extended to include four-wheeled vehicles, two-wheeled vehicles, pedestrians, and traffic lights, etc. The categories of line segments (semantic segmentation) include drivable areas and lane lines.
[0046] Embodiment 4: Block-based line segment detection method
[0047] When training line segment detection, the data needs to be divided into different blocks. The size of the block depends on the input of the architecture and the number of downsampling. In this embodiment, the architecture is the output of the detection head after 5 downsampling, and the width and height input of the architecture is 640×640, then the width and height of the block are That is, a block with a width and height of 20×20, such as Figure 4 The output dimension of the line segment detection head of this architecture is [n,h,w,l], where n represents the maximum number of line segments that can appear in a block, h and w represent the height and width of the feature map output by the lane line detection head (Head), and l represents the confidence of the line segment, the upper and lower limits of the line segment, and the polynomial coefficients of the line segment.
[0048] When training line segment detection, the data needs to be divided into different blocks. The size of the block depends on the input of the architecture and the number of downsampling. In this embodiment, the architecture is the output of the detection head after 5 downsampling, and the width and height input of the architecture is 640×640, then the width and height of the block are That is, a block with a width and height of 20×20, such as Figure 4 The output dimension of the line segment detection head of this architecture is [n,h,w,l], where n represents the maximum number of line segments that can appear in a block, h and w represent the height and width of the feature map output by the lane line detection head (Head), and l represents the confidence of the line segment, the upper and lower limits of the line segment, and the polynomial coefficients of the line segment.
[0049] Line segment detection loss function loss ), as shown in Table 1, which includes the loss of confidence (c loss ), category loss (cl loss ), the upper limit loss of the line segment (up loss ) and lower limit loss (lo loss ), and the loss of the line segment (x loss ), the loss is the predicted value obtained by the polynomial estimation model to obtain a set of x values, and then compared with the correct result of the data set (GroundTrue), and then the MSE (Mean Squared Error) method is performed. The two values are subtracted and squared, and the sum is taken to obtain the average value as the basis for loss regression. Then the MSE loss regression is performed with GT.
[0050] Table 1
[0051]
[0052] The loss function used in target detection (Obj loss), as shown in Table 2, like YOLO v5, the loss function of target detection includes the loss of category confidence Loss of object confidence and the loss for object box regression (CIOU).
[0053] Table 2
[0054]
[0055] The "object frame" referred to in the present invention refers to the result output by object detection and identification. The "real frame" referred to in the present invention refers to the real frame in step S1 of the present invention, because a training data set needs to be prepared when training the network architecture. The "prediction frame" referred to in the present invention refers to the object detection information output in step S5 of the present invention, which is represented as the prediction frame. The loss function of the object frame regression adopts CIOU Loss (Complete IOU Loss), as shown in Table 3. The loss function takes into account the distance, overlap rate, scale and aspect ratio between the real frame and the prediction frame, such as Figure 5 As shown, the target box regression can be more stable. The CIOU algorithm is shown in formula (10) in Table 3, where α is a weight function, and the size can be controlled by the overlapping area, as shown in formula (11), and v is used to measure the proportional consistency between the real box and the predicted box, as shown in formula (12) in Table 3.
[0056] Table 3
[0057]
[0058] Example 5: Evaluation method for object detection
[0059] The object detection evaluation method uses the intersection over union (IoU) for evaluation, which is a measure of the degree of overlap between the predicted bounding box and the annotated data bounding box. Its definition is shown in formula (13) in Table 4. The overlapping area of the predicted bounding box and the annotated data bounding box is divided by the set association area of the predicted bounding box and the annotated data bounding box, as shown in Figure 6 shown.
[0060] Table 4
[0061]
[0062] Example 6: Evaluation method for line segment detection
[0063] The term "TuSimple" in this invention refers to a mainstream data set provider for lane line detection, and its data set also provides a method for calculating the accuracy, as shown in Formula 14 in Table 5. The evaluation method for line segment detection uses the indicators of TuSimple for evaluation, as shown in Formula (14) in Table 5, where C clip is the total number of correctly predicted points, S clip is the total number of actually annotated points. Based on these experimental settings, for target detection, the correctness of the detection result is defined based on the IoU threshold of 0.5, and the correctness of line segment detection is sampled. The predicted point is within 20 pixels of the corresponding label point, which is the correct predicted point.
[0064] Table 5
[0065]
[0066] After the line segment detection, many similar line segments will appear on the same block at the same time, such as Figure 7 As shown on the left, these similar line segments can be filtered by non-maximum suppression. The filtering method has three steps. The first step is to sort all line segments from high to low confidence, and select the line segment with the highest confidence as the target. The second step is to select the lowest upper and lower limits of the two line segments, align the line segments, and then filter through the normalized value of cosine similarity to filter all line segments above the set threshold. The third step is to select the second highest confidence line segment as the target, and repeat steps one and two until all line segments are filtered. Finally, the following is obtained. Figure 7 The block line segment shown on the right, Figure 8 This is a result diagram of this embodiment.
[0067] The key to the present invention is to directly merge semantic segmentation and object detection into the same network architecture. Since there are many types of detection for object detection and semantic segmentation, the object detection in this embodiment is based on four-wheel vehicle detection, and the semantic segmentation is based on lane line detection. The above two detection categories are used as examples to illustrate that they can be successfully implemented. The implementation results of the accuracy and performance comparison of the present invention are shown in Table 6. The average accuracy can be obtained by only decreasing by about 0.1%, while the inference time is greatly reduced by about 47.9%, and the overall cost performance is about 479 times.
[0068] Table 6
[0069]
[0070] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. A person having ordinary knowledge in the technical field to which the present invention belongs may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the scope of the attached patent application.
Claims
1. A driving assistance system, characterized in that: The system utilizes a deep neural network architecture for performing object detection and semantic segmentation detection and recognition. The system includes: an image capture module configured to capture an image; An image cutting module configured to cut the image into a plurality of block images; a processing module configured to execute the deep neural network architecture according to the block images to construct a plurality of common feature maps for object detection and semantic segmentation, a plurality of object boxes and a plurality of block line segment parameters, and configured to perform filtering and merging according to the block line segment parameters; An output module is configured to output information of object detection and detection semantic segmentation.
2. The system of claim 1, wherein the image acquisition module comprises at least one RGB camera.
3. The system according to claim 1, wherein the processing module comprises an object detection module and a line segment detection module, the object detection module is configured to execute an object detection evaluation method, the object detection evaluation method comprises using an intersection over union (IoU) method for evaluation, and the line segment detection module is configured to execute a line segment detection evaluation method, the line segment detection evaluation method comprises using Tusimple indicators for evaluation.
4. The system of claim 1, wherein the deep neural network architecture comprises implementing an algorithm comprising a loss function for line segment detection, a loss function used in target detection, and a loss function for object box regression.
5. A driving assistance calculation method, characterized in that: The method utilizes a deep neural network architecture, which is used to perform object detection and semantic segmentation detection and recognition. The method includes the following steps: Step S1: capturing an image; Step S2: dividing the image into a plurality of block images; Step S3: executing the deep neural network architecture according to the block images to construct a plurality of common feature maps for object detection and semantic segmentation, a plurality of object boxes and a plurality of block line segment parameters; Step S4: performing filtering and merging according to the block line segment parameters; Step S5: Output object detection and semantic segmentation information. 6 . The method according to claim 5 , wherein step S1 is to obtain the image through at least one RGB camera.
7. The method according to claim 5, wherein step S3 comprises executing an object detection evaluation method and a line segment detection evaluation method, wherein the object detection evaluation method comprises using an Intersection Over Union (IoU) method for evaluation, and the line segment detection evaluation method comprises using a Tusimple indicator for evaluation.
8. The method according to claim 5, wherein step S3 comprises implementing an algorithm, wherein the algorithm comprises a loss function for line segment detection, a loss function used in target detection, and a loss function for object box regression.
