Automatic driving image recognition method based on region-of-interest scheduling

Abstract

The invention discloses an automatic driving image recognition method based on region-of-interest scheduling. The method comprises the following steps: initializing processing time and recognition precision of images with different sizes; acquiring original data of a camera and a laser radar, and segmenting a region containing an obstacle in an image into a plurality of regions of interest; performing key evaluation on the region of interest based on the distance and the speed acquired by the laser radar; taking the region of interest as an independent image recognition task, adding the task into a scheduling queue according to the sequence of collision time from small to large, and ensuring that the recognition time of the task in the queue does not exceed a task deadline through task size compression; inputting the tasks in the queue into a ResNet network model in batches, and outputting a category label and a confidence coefficient score of an obstacle and a local bounding box coordinate under a current region-of-interest coordinate system; global absolute coordinates of the obstacle in the original complete image are calculated and transmitted to a decision planning module of the automatic driving system to serve as a basis for path planning and obstacle avoidance control.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology and relates to an image recognition method for autonomous driving based on region of interest scheduling. Background Technology

[0002] In the field of autonomous driving, the ability to perceive the vehicle's surrounding environment is crucial for ensuring driving safety. Currently, mainstream perception technologies primarily rely on visual perception, capturing images through cameras and using image recognition technology to identify obstacles such as pedestrians, vehicles, and traffic signs. The core of these technologies is achieving high-precision and real-time recognition of objects in images.

[0003] Currently, many autonomous driving systems employ end-to-end visual perception solutions. These solutions typically input complete images captured by cameras into deep learning models, such as convolutional neural networks (CNNs) and deep neural networks (DNNs), for object detection or semantic segmentation. Vehicle cameras continuously capture color images of the area in front of the vehicle at high frame rates (typically 30 frames per second or higher). These images serve as the raw data for the autonomous driving system's environmental perception. The acquired complete, high-resolution images are then directly input into a pre-trained deep learning model. After processing, the model outputs recognition results. These results are subsequently used by the autonomous driving decision-making module, for example, to adjust the vehicle's speed based on the position and speed of vehicles ahead, or to change driving strategies based on traffic signs.

[0004] The YOLO algorithm employs an end-to-end, single-stage approach, using the entire image as input. The image undergoes only one forward propagation in the neural network to simultaneously predict the category, location, and confidence score of all target objects in the image. Its high efficiency has made it a mainstream solution for real-time object detection. Furthermore, YOLO achieves a balance between speed and accuracy through continuous iteration and optimization of the network structure and loss function, significantly improving the real-time processing capabilities of autonomous driving systems. While end-to-end image recognition methods are simple and efficient, achieving high-precision recognition typically requires processing high-resolution images. However, processing the entire image in-depth requires substantial computational resources and powerful onboard computing units, leading to high costs and power consumption for vehicle systems. Additionally, this method treats all pixels in the image equally, expending significant computational resources on areas with little impact on driving decisions, such as the sky, distant buildings, and roadside trees. This indiscriminate processing results in wasted resources.

[0005] Patent CN107944351B provides a technical solution to address the problem of excessive computation in deep learning image recognition. While it effectively reduces computation by lowering the resolution of non-interest regions (NIRs), it still has limitations. Although this method improves efficiency by reducing computation, it does not propose an effective scheduling algorithm to ensure that the identification of critical obstacles is completed before a strict deadline. In autonomous driving, real-time recognition is a key safety factor, and simply reducing computation cannot fully guarantee this. Furthermore, this method cannot dynamically adjust the priority of different regions. Simply dividing the area in front of the vehicle into NIRs and other areas into NIRs cannot handle obstacles that suddenly appear in complex and changing driving scenarios, such as vehicles or pedestrians suddenly appearing from the left. This method does not provide a mechanism to dynamically adjust the recognition priority and priority allocation of NIRs. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention aims to provide an autonomous driving image recognition method based on region of interest scheduling, thereby resolving the issues of existing autonomous driving visual perception technologies struggling to balance real-time performance and recognition accuracy, as well as the unreasonable allocation of computing resources.

[0007] This invention provides an image recognition method for autonomous driving based on region of interest scheduling, comprising:

[0008] Step 1: Pre-train the ResNet network model and preload the model into the computation unit;

[0009] Step 2: Initialize the processing time and corresponding recognition accuracy for images of different sizes, and determine the minimum recognition accuracy requirement for the region of interest in the image;

[0010] Step 3: Acquire raw data from the camera and LiDAR in real time, and use the precise point cloud data provided by the LiDAR to accurately segment the area containing potential obstacles in the image into multiple regions of interest;

[0011] Step 4: Based on the distance and velocity obtained by the lidar, calculate the collision time of the obstacles corresponding to the region of interest, and perform a critical assessment of each region of interest based on the collision time;

[0012] Step 5: Treat the region of interest as an independent image recognition task, add the recognition tasks to the scheduling queue in ascending order of collision time, and ensure that the recognition time of the tasks in the scheduling queue does not exceed the task deadline through a task size compression mechanism.

[0013] Step 6: Input the tasks in the scheduling queue into the ResNet network model in batches, and output the obstacle category label, confidence score, and local bounding box coordinates in the current region of interest coordinate system concurrently;

[0014] Step 7: Obtain the cropping start point coordinates of each region of interest in the original image and the compression factor of each region of interest, and calculate the global absolute coordinates of the obstacle in the original complete image;

[0015] Step 8: Transmit the global absolute coordinates to the decision planning module of the autonomous driving system as the input for path planning and obstacle avoidance control.

[0016] The autonomous driving image recognition method based on region of interest scheduling of the present invention has the following beneficial effects:

[0017] (1) This invention significantly reduces the total computational load of image recognition by downgrading the resolution of non-critical areas and dynamically scheduling tasks, thereby greatly improving processing efficiency. Under the same hardware conditions, a higher frame rate (Frames Per Second, FPS) can be achieved, ensuring that the system can meet the strict real-time requirements of autonomous driving.

[0018] (2) Existing technologies treat all obstacles equally. When computing resources are scarce, the recognition accuracy of critical obstacles may be forced to decrease, and there may even be cases where the execution time of critical obstacle recognition tasks exceeds the specified deadline. This invention, through priority division and dynamic scheduling, ensures that critical obstacles that are crucial to driving safety (such as pedestrians and vehicles at close range) are always processed with high resolution and in a timely manner, thereby ensuring that their recognition accuracy is not affected and that the recognition completion time meets the deadline requirements.

[0019] (3) Existing mainstream methods distribute computing resources evenly across the entire image, leading to overcomputation of non-critical information (such as distant backgrounds) and causing serious waste of resources. This invention fundamentally optimizes the utilization of computing resources by concentrating resources on critical areas and downgrading non-critical areas.

[0020] (4) When encountering complex scenarios or sudden situations (such as suddenly appearing obstacles), the scheduling algorithm of the present invention can dynamically allocate processing resources to newly emerging critical tasks and ensure that they are completed within the deadline, avoiding system delays or crashes caused by task queue congestion. Attached Figure Description

[0021] Figure 1 This is a flowchart of an autonomous driving image recognition method based on region of interest scheduling according to the present invention. Detailed Implementation

[0022] like Figure 1 As shown, an autonomous driving image recognition method based on region of interest scheduling according to the present invention includes:

[0023] Step 1: Pre-train the ResNet network model and preload the model into the computation unit.

[0024] Step 2: Initialize the processing time and corresponding recognition accuracy for images of different sizes, and determine the minimum recognition accuracy requirement for the region of interest in the image, specifically:

[0025] Step 2.1: Based on the performance of the pre-trained ResNet network model, determine the processing time and corresponding recognition accuracy for several different image sizes (e.g., 32×32, 64×64, 128×128, 512×512).

[0026] Step 2.2: Store the processing time and recognition accuracy in a lookup table for subsequent fast lookups and calculations.

[0027] Step 2.3: Based on the requirements of the autonomous driving scenario, determine the minimum recognition accuracy requirements for key areas to ensure the real-time performance and accuracy of key obstacle recognition even when resources cannot meet the requirements for completing all image recognition tasks.

[0028] Step 3: Acquire raw data from the camera and LiDAR in real time. Using the precise point cloud data provided by the LiDAR, accurately segment the areas containing potential obstacles in the image into multiple regions of interest, specifically:

[0029] Step 3.1: Perform ground filtering on the raw 3D point cloud data collected by the lidar to remove ground data that is not useful for obstacle identification.

[0030] Step 3.2: For the remaining non-ground 3D point cloud data, use the Euclidean clustering algorithm to perform clustering processing, dividing the spatially adjacent non-ground 3D point cloud data into independent obstacle point cloud clusters, with each point cloud cluster representing a potential physical obstacle.

[0031] Step 3.3: Through coordinate system transformation, transform the coordinates of each point in the point cloud cluster from the radar coordinate system to the camera coordinate system. Let the point in the radar coordinate system be... The point in the camera coordinate system is The conversion formula is:

[0032]

[0033] Where R is a 3×3 rotation matrix and T is a 3×1 translation matrix, both of which are obtained by joint calibration of the lidar and the camera.

[0034] Step 3.4: Project the 3D points in the camera coordinate system onto the 2D image plane using the camera's built-in intrinsic parameter matrix.

[0035] Step 3.5: For each obstacle point cloud cluster, project all its points onto the image plane, and then divide the initial bounding box in the image based on the maximum and minimum x-coordinates, maximum and minimum y-coordinates of the projected point set on the image coordinate axes.

[0036] Step 3.6: In order to prevent the obstacle edge from being truncated due to sparse radar point cloud or projection error, the initial bounding box is expanded outward to the closest size image size among the several sizes set in Step 2. The finally cropped area is the region of interest of the obstacle.

[0037] Step 4: Based on the distance and velocity acquired by the lidar, calculate the collision time of the obstacles corresponding to the region of interest. Perform a critical evaluation for each region of interest based on the collision time, specifically:

[0038] Step 4.1: For each segmented region of interest, calculate the coordinates of the geometric center or nearest point of the point cloud cluster in the vehicle's direction of travel, and determine these coordinates as the current relative distance to the obstacle. .

[0039] Step 4.2: Determine the time interval between two adjacent frames of point cloud data acquisition based on the preset acquisition frequency f of the lidar. Based on the relative distance to the same target obstacle at the previous acquisition time Calculate the distance difference between the current time and the previous time. ,Right now .

[0040] Step 4.3: Calculate the vehicle's speed relative to the obstacle based on the distance difference and the time interval between the acquisition of two adjacent frames of point cloud data.

[0041]

[0042] Step 4.4: Calculate the collision time of obstacles in the region of interest according to the following formula. Only considering Collision time of obstacles in the region of interest that is negative:

[0043]

[0044] Step 4.5: Based on the set collision safety time threshold With collision time A comparison was made, and a critical assessment was performed on each region of interest. When, mark the region of interest as a key obstacle, when When this happens, the region of interest is marked as a non-critical obstacle.

[0045] Step 5: Treat the region of interest as an independent image recognition task, add the recognition tasks to the scheduling queue in ascending order of collision time, and ensure that the recognition time of the tasks in the scheduling queue does not exceed the task deadline through a task size compression mechanism. Specifically:

[0046] Step 5.1: Add the identification tasks of critical obstacles and non-critical obstacles to the scheduling queue in order of collision time from smallest to largest. When each task is added to the queue, calculate the total identification processing time of all critical tasks in the current queue based on the processing time of different sized images initialized in Step 2.

[0047] Step 5.2: Compare the total recognition processing time with the task deadline. If the total recognition processing time exceeds the task deadline, stop adding tasks to the scheduling queue and start compressing the image size from the end of the queue until the total recognition processing time meets the task deadline. Then, continue adding tasks to the scheduling queue. Specifically:

[0048] Step 5.2.1: Set the vehicle image sampling period to the task deadline. When the total recognition processing time exceeds the task deadline, select the task at the end of the current queue and determine whether it is a critical obstacle recognition task or a non-critical obstacle recognition task based on the criticality assessment result. If it is determined to be a non-critical obstacle recognition task, proceed to step 5.2.2; otherwise, proceed to step 5.2.5.

[0049] Step 5.2.2: Compress the image size to the preset minimum resolution, then recalculate the total recognition processing time for all tasks in the current queue. If the total recognition processing time meets the task deadline, continue to add tasks to the scheduling queue.

[0050] Step 5.2.3: If the total recognition processing time exceeds the task deadline, continue to select the next task with the longest collision time, and determine whether it is a critical obstacle recognition task or a non-critical obstacle recognition task.

[0051] Step 5.2.4: If the task is determined to be a non-critical obstacle identification task, repeat steps 5.2.2 and 5.2.3; otherwise, proceed to step 5.2.5.

[0052] Step 5.2.5: When the task is determined to be the identification of a key obstacle, the image is compressed step by step according to the preset image resolution from large to small, and the input tensor size is reduced to the next step using a bilinear interpolation algorithm.

[0053] Step 5.2.6: Recalculate the total recognition and processing time for all tasks in the current queue. If the total recognition and processing time meets the task deadline, continue to add tasks to the scheduling queue.

[0054] Step 5.2.7: If the task deadline has passed, determine whether the image size of the current task has been compressed to the lowest possible resolution.

[0055] Step 5.2.8: If the resolution is not compressed to the minimum, repeat steps 5.2.5 to 5.2.7 until the resolution is compressed to the minimum; otherwise, proceed to step 5.2.9.

[0056] Step 5.2.9: Continue to select the next critical obstacle identification task with the largest collision time, and repeat steps 5.2.5 to 5.2.8.

[0057] Step 5.3: If the image is compressed to the set minimum size and still cannot meet the deadline requirement, the remaining non-critical tasks will be discarded.

[0058] Step 6: Input the tasks in the scheduling queue into the ResNet network model in batches, and concurrently output the obstacle category labels, confidence scores, and local bounding box coordinates in the current region of interest coordinate system. Specifically:

[0059] Step 6.1: Group the tasks in the scheduling queue according to the image size to generate several independent scheduling batches.

[0060] Step 6.2: Feed the region of interest (ROI) images into the pre-trained ResNet network model in batches. Utilizing the parallel computing power of the GPU, the model simultaneously extracts features from multiple ROIs within each batch and concurrently outputs the local recognition results within each ROI. These results include the obstacle's category label, confidence score, and local bounding box coordinates in the current ROI image coordinate system. Category labels include obstacles such as vehicles, pedestrians, roadblocks, and cyclists.

[0061] Step 7: Since the bounding boxes output by the neural network are based on the cropped and compressed local images of the regions of interest, the system must restore them to the original coordinate system from the vehicle's perspective. Obtain the cropping start point coordinates of each region of interest in the original image and the scaling factor for each region of interest, and calculate the global absolute coordinates of the obstacles in the original complete image;

[0062] Step 8: Transmit the global absolute coordinates to the decision planning module of the autonomous driving system as the input for path planning and obstacle avoidance control.

[0063] Previous image recognition algorithms treated all regions in an image equally, performing indiscriminate calculations. This resulted in a significant waste of computational resources on non-critical areas (such as the sky or distant trees) that had little impact on driving decisions. This invention, through multi-sensor fusion (radar data + image), not only extracts obstacle regions as regions of interest within the image but also intelligently divides the image into critical and non-critical obstacle regions. This fundamentally changes the allocation logic of computational resources, ensuring that limited computing power is concentrated on the critical areas requiring the highest precision for identification, thereby maximizing resource utilization.

[0064] Existing algorithms are limited by model complexity and image size, making dynamic adjustment impossible in emergency situations. When the workload is heavy or computational resources are scarce, it cannot guarantee that the identification results for critical obstacles will be returned within the strict deadline. To ensure high-risk targets are processed first, this invention eliminates the traditional first-in-first-out (FIFO) mechanism of GPU task processing. Instead, it implements a reordering mechanism based on criticality and priority. This mechanism dynamically evaluates the priority and processing time of each identification task and proactively compresses tasks to ensure that the identification of critical obstacles is prioritized and meets real-time requirements (deadlines). This represents a fundamental shift from passive execution to proactive safeguarding, providing crucial safety redundancy for autonomous driving.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image recognition method for autonomous driving based on region of interest scheduling, characterized in that, include: Step 1: Pre-train the ResNet network model and preload the model into the computation unit; Step 2: Initialize the processing time and corresponding recognition accuracy for images of different sizes, and determine the minimum recognition accuracy requirement for the region of interest in the image; Step 3: Acquire raw data from the camera and LiDAR in real time, and use the precise point cloud data provided by the LiDAR to accurately segment the area containing potential obstacles in the image into multiple regions of interest; Step 4: Based on the distance and velocity obtained by the lidar, calculate the collision time of the obstacles corresponding to the region of interest, and perform a critical assessment of each region of interest based on the collision time; Step 5: Treat the region of interest as an independent image recognition task, add the recognition tasks to the scheduling queue in ascending order of collision time, and ensure that the recognition time of the tasks in the scheduling queue does not exceed the task deadline through a task size compression mechanism. Step 6: Input the tasks in the scheduling queue into the ResNet network model in batches, and output the obstacle category label, confidence score, and local bounding box coordinates in the current region of interest coordinate system concurrently; Step 7: Obtain the cropping start point coordinates of each region of interest in the original image and the scaling factor of each region of interest, and calculate the global absolute coordinates of the obstacle in the original complete image; Step 8: Transmit the global absolute coordinates to the decision planning module of the autonomous driving system as the input for path planning and obstacle avoidance control.

2. The autonomous driving image recognition method based on region of interest scheduling according to claim 1, characterized in that, Step 2 specifically involves: Step 2.1: Based on the performance of the pre-trained ResNet network model, determine the processing time and corresponding recognition accuracy for several images of different sizes; Step 2.2: Store the processing time and recognition accuracy in a lookup table for subsequent fast lookups and calculations; Step 2.3: Based on the requirements of the autonomous driving scenario, determine the minimum recognition accuracy requirements for key areas to ensure the real-time performance and accuracy of key obstacle recognition even when resources cannot meet the requirements for completing all image recognition tasks.

3. The autonomous driving image recognition method based on region of interest scheduling according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Perform ground filtering on the raw 3D point cloud data collected by the lidar to remove ground data that is not useful for obstacle identification; Step 3.2: For the remaining non-ground 3D point cloud data, use the Euclidean clustering algorithm to perform clustering processing, divide the spatially adjacent non-ground 3D point cloud data into independent obstacle point cloud clusters, and each point cloud cluster represents a potential physical obstacle; Step 3.3: Through coordinate system transformation, transform the coordinates of each point in the point cloud cluster from the radar coordinate system to the camera coordinate system. Let the point in the radar coordinate system be... The point in the camera coordinate system is The conversion formula is: Where R is a 3×3 rotation matrix and T is a 3×1 translation matrix, both of which are obtained by joint calibration of the lidar and the camera; Step 3.4: Project the 3D points in the camera coordinate system onto the 2D image plane using the camera's built-in intrinsic parameter matrix; Step 3.5: For each obstacle point cloud cluster, after projecting all its points onto the image plane, divide the initial bounding box in the image according to the maximum and minimum x-coordinates, maximum and minimum y-coordinates of the projected point set on the image coordinate axes. Step 3.6: In order to prevent the obstacle edge from being truncated due to sparse radar point cloud or projection error, the initial bounding box is expanded outward to the closest size image size among the several sizes set in Step 2. The finally cropped area is the region of interest of the obstacle.

4. The autonomous driving image recognition method based on region of interest scheduling according to claim 1, characterized in that, Step 4 specifically involves: Step 4.1: For each segmented region of interest, calculate the coordinates of the geometric center or nearest point of the point cloud cluster in the vehicle's direction of travel, and determine these coordinates as the current relative distance to the obstacle. ; Step 4.2: Determine the time interval between two adjacent frames of point cloud data acquisition based on the preset acquisition frequency f of the lidar. Based on the relative distance to the same target obstacle at the previous acquisition time Calculate the distance difference between the current time and the previous time. ,Right now ; Step 4.3: Calculate the vehicle's speed relative to the obstacle based on the distance difference and the time interval between the acquisition of two adjacent frames of point cloud data. Step 4.4: Calculate the collision time of obstacles in the region of interest according to the following formula. Only considering Collision time of obstacles in the region of interest that is negative: Step 4.5: Based on the set collision safety time threshold With collision time A comparison was made, and a critical assessment was performed on each region of interest. When, mark the region of interest as a key obstacle, when When this happens, the region of interest is marked as a non-critical obstacle.

5. The autonomous driving image recognition method based on region of interest scheduling according to claim 1, characterized in that, Step 5 specifically involves: Step 5.1: Add the identification tasks of critical obstacles and non-critical obstacles to the scheduling queue in order of collision time from smallest to largest. When each task is added to the queue, calculate the total identification processing time of all critical tasks in the current queue based on the processing time of images of different sizes initialized in Step 2. Step 5.2: Compare the total recognition processing time with the task deadline. When the total recognition processing time exceeds the task deadline, stop adding tasks to the scheduling queue and start compressing the image size from the end of the queue one by one until the total recognition processing time meets the task deadline. Then continue adding tasks to the scheduling queue. Step 5.3: If the deadline requirement is still not met when the image is compressed to the set minimum size, the remaining non-critical tasks will be discarded.

6. The autonomous driving image recognition method based on region of interest scheduling according to claim 5, characterized in that, Step 5.2 specifically involves: Step 5.2.1: Set the vehicle image sampling period to the task deadline. When the total recognition processing time exceeds the task deadline, select the task at the end of the current queue and determine whether it is a critical obstacle recognition task or a non-critical obstacle recognition task based on the criticality assessment result. If it is determined to be a non-critical obstacle recognition task, proceed to step 5.2.2; otherwise, proceed to step 5.2.

5. Step 5.2.2: Compress the image size to the preset minimum resolution, then recalculate the total recognition processing time for all tasks in the current queue. If the total recognition processing time meets the task deadline, continue to add tasks to the scheduling queue. Step 5.2.3: If the total recognition processing time exceeds the task deadline, continue to select the next task with the longest collision time, and determine whether it is a critical obstacle recognition task or a non-critical obstacle recognition task. Step 5.2.4: If the task is determined to be a non-critical obstacle identification task, repeat steps 5.2.2 and 5.2.3; otherwise, proceed to step 5.2.

5. Step 5.2.5: When the task is determined to be the identification of a key obstacle, the image is compressed step by step according to the preset image resolution from large to small, and the input tensor size is reduced to the next step using the bilinear interpolation algorithm. Step 5.2.6: Recalculate the total recognition and processing time for all tasks in the current queue. If the total recognition and processing time meets the task deadline, continue to add tasks to the scheduling queue. Step 5.2.7: If the task deadline has passed, determine whether the image size of the current task has been compressed to the lowest possible resolution; Step 5.2.8: If the resolution is not compressed to the minimum, repeat steps 5.2.5 to 5.2.7 until the resolution is compressed to the minimum; otherwise, proceed to step 5.2.

9. Step 5.2.9: Continue to select the next critical obstacle identification task with the largest collision time, and repeat steps 5.2.5 to 5.2.

8.

7. The autonomous driving image recognition method based on region of interest scheduling according to claim 1, characterized in that, Step 6 specifically involves: Step 6.1: Group the tasks in the scheduling queue according to image size to generate several independent scheduling batches; Step 6.2: Feed the region of interest images into the pre-trained ResNet network model in batches. Utilize the parallel computing power of the GPU, the model extracts features from multiple regions of interest in the batch at the same time and outputs the local recognition results within each region of interest concurrently. The results include the category label of the obstacle, the confidence score, and the local bounding box coordinates in the current region of interest image coordinate system.

Images