Automatic driving target detection method based on gradient re-parameterization and center point estimation
By using gradient re-parsing network and center point estimation detection algorithm in the autonomous driving target detection technology, the problems of low detection accuracy, large computing resource consumption and complex post-processing in the existing technology are solved, and efficient autonomous driving target detection is achieved , meeting the real-time detection needs in autonomous driving scenarios.
Patent Information
- Application Number
- CN202411133027.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2025-02-07
AI Technical Summary
When the existing autonomous driving target detection technology deals with multi-scale, aspect ratio and position changes, there are problems such as low detection accuracy, large computing resources consumption, and complex post-processing, which is difficult to meet the real-time detection requirements in autonomous driving scenarios.
Using the automatic driving target detection method based on gradient reparameterization and center point estimation, the image is characterized by the gradient reparameterization network to obtain a multi-scale feature map, and the thermal map is detected through the center point estimation detection algorithm to achieve the goal. Output of bounding boxes, categories, and confidence.
This method can achieve efficient parallelization in the training and inference process, reduce computing resource requirements, reduce post-processing steps, achieve a balance of accuracy and speed, and meet the real-time detection requirements of obstacles in autonomous driving scenarios.
Smart Images
Figure CN119399422A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of autonomous driving target detection, and specifically relates to an autonomous driving target detection method based on gradient reparameterization and center point estimation. Background Art
[0002] Object detection aims to identify all instances of predefined categories in an image and achieve object localization and classification. It is a very important and challenging visual task. In recent years, with the rapid development of deep learning, object detection technology has made new breakthroughs and has been widely used in many fields in the real world, such as autonomous driving.
[0003] Existing target detection technologies are generally divided into two-stage detection and one-stage detection. Two-stage detection first generates a large number of candidate boxes, and then predicts and classifies the candidate boxes. This method has high detection accuracy, but is slow and has high model complexity. It is rarely used in engineering, especially in the field of autonomous driving with high real-time requirements. In contrast, the one-step, end-to-end one-stage detection method has a high detection speed and can meet the deployment of mobile devices and edge devices, but its accuracy is limited, especially for small and dense targets.
[0004] Objects in autonomous driving scenarios often have changes in scale, aspect ratio, position, etc. The above-mentioned candidate frame-based detection method needs to design several reference frames to match environmental features and rely on post-processing to remove redundant and low-confidence frames. It is prone to problems such as imbalance between positive and negative samples, a large number of hyper-parameter settings, and slow convergence.
[0005] Although many target detection algorithms have excellent accuracy and speed, they rely on a lot of computing resources and memory overhead, so they cannot be applied to low-power mobile / edge devices. Components such as depthwise separable convolution and channel shuffling can reduce network complexity while maintaining accuracy, but actually increase memory access costs and computational costs. The structural reparameterized network, that is, the multi-branch structure of the decoupling training period is a direct connection structure of the inference period, which can achieve a compromise between speed and accuracy. But fundamentally, its complex training period structure still consumes too much time and memory, which is not very friendly to scenarios with limited computing resources, urgent delivery or fast iteration of models, and there is a serious problem of accuracy loss after quantization. Summary of the invention
[0006] The purpose of the present invention is to provide an autonomous driving target detection method based on gradient reparameterization and center point estimation, which has efficient training and reasoning processes, a high degree of parallelization, is quantization-friendly, does not rely on post-processing, and can achieve a balance between accuracy and speed to meet the real-time detection requirements of obstacles in autonomous driving scenarios.
[0007] In order to solve the above technical problems, the technical solution of the present invention is: an automatic driving target detection method based on gradient reparameterization and center point estimation, comprising:
[0008] In autonomous driving, the collected images are preprocessed;
[0009] The preprocessed image is processed by gradient reparameterization network to obtain a multi-scale feature map;
[0010] Perform feature fusion on multi-scale feature maps to obtain a heat map;
[0011] The heat map is used to detect targets through the center point estimation detection algorithm to obtain an output image containing the target's bounding box, category, and confidence level.
[0012] The specific method of performing feature processing on the preprocessed image through the gradient reparameterization network to obtain a multi-scale feature map is:
[0013] Construct a gradient reparameterized network for feature processing: perform hyperparameter search on the preset data set, obtain the scale value for optimizing the gradient and extract the constant multiplier from the scale value; create an optimizer for training, multiply the original gradient by the constant multiplier to obtain the optimized gradient, update the optimizer with the optimized gradient, introduce the VGG model that only contains a 3x3 convolution operator, and combine the VGG model and the updated optimizer to obtain the gradient reparameterized network;
[0014] The gradient reparameterization network is used to perform forward propagation, backward propagation, and parameter update on the preprocessed image to obtain a multi-scale feature map.
[0015] The specific method of fusing the multi-scale feature map to obtain the heat map is as follows: increasing the network's receptive field through spatial pyramid pooling SPP and separating multiple features of the multi-scale feature map; fusing the multi-scale feature map through the feature pyramid network FPN and the path aggregation network PAN, and outputting the heat map.
[0016] The method of performing target detection on the heat map through the center point estimation detection algorithm to obtain the output image containing the target's bounding box, category and confidence is as follows:
[0017] Perform three upsampling operations on the heat map to reduce the resolution of its feature map;
[0018] Add three head structures to the heat map after the resolution is reduced, which are used to predict the center point heat map, center point coordinate compensation and bounding box size respectively, and determine the category of the heat map target according to the predicted center point heat map, center point coordinate compensation and bounding box size;
[0019] Predict the center point and confidence of each target of each category based on the center point heat map. The center point of the target is the peak of the heat map, and regress the center point coordinate compensation and the width and height of the bounding box.
[0020] Assume that the coordinates of the center point of a target are (x i ,y i ), coordinate compensation is (δx i ,δy i ), width and height are (w i ,h i ), then the coordinates of the upper left corner and the lower right corner of the target bounding box are represented as:
[0021] Coordinates of the upper left corner: (x i +δx i -w i / 2,y i +δy i -h i / 2)
[0022] Coordinates of the lower right corner: (x i +δx i +w i / 2,y i +δy i +h i / 2)
[0023] The bounding box of the target is determined based on the coordinates of the upper left corner and the lower right corner.
[0024] Preprocessing includes at least color space enhancement, image size conversion and pixel normalization.
[0025] Color space enhancement includes at least contrast, saturation, hue, and blur.
[0026] Image resizing is a scaling strategy that keeps the aspect ratio unchanged. When the input image does not meet the required aspect ratio, the low-information area is removed, the bottom of the image is cropped, and the cropped image is scaled to the required size using the nearest neighbor or bilinear sampling method.
[0027] Pixel normalization is used to make the input image data have the same distribution. Z-score normalization is used, that is, the distribution with a mean of 0 and a standard deviation of 1. The formula is:
[0028]
[0029] Among them, z is the image data after pixel normalization, x is the acquired image, μ is the mean of the data in the preset data set, and σ is the standard deviation of the data in the preset data set.
[0030] A target detection system for autonomous driving based on gradient reparameterization and center point estimation is also provided, comprising: a preprocessing module, a feature extraction module, a feature fusion module and a target detection module; wherein,
[0031] A preprocessing module, used to preprocess images collected during autonomous driving;
[0032] A feature extraction module is used to perform feature processing on the preprocessed image through a gradient reparameterization network to obtain a multi-scale feature map;
[0033] Feature fusion module, used to fuse multi-scale feature maps to obtain heat maps;
[0034] The target detection module is used to perform target detection on the heat map through the center point estimation detection algorithm to obtain an output image containing the target's bounding box, category and confidence.
[0035] The fisheye image acquired by the vehicle-mounted fisheye camera is used as the image of the input preprocessing module.
[0036] Compared with the prior art, the present invention has the following beneficial effects:
[0037] The detection method based on the gradient reparameterization network and the center point of the present invention is simple and efficient, does not require post-processing, and can adaptively detect objects of different scales, aspect ratios and positions; the algorithm based on the gradient reparameterization of the present invention eliminates the need for network structure conversion, has the advantage of being quantization-friendly, and meets the deployment requirements of autonomous driving edge devices; the present invention has the dual efficiency of training and reasoning, and is more beneficial for fields with limited computing resources, urgent delivery, and rapid model iteration. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a schematic diagram of the process of the present invention;
[0039] Figure 2 FIG. 2 is a schematic diagram of the working example of the present invention. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0041] The technical solution of the present invention is:
[0042] Embodiment 1:
[0043] Autonomous driving target detection method based on gradient reparameterization and center point estimation, such as Figure 1 As shown, including:
[0044] S1. Preprocessing of collected images during autonomous driving;
[0045] S2, feature processing is performed on the preprocessed image through a gradient reparameterization network to obtain a multi-scale feature map;
[0046] S3, perform feature fusion on multi-scale feature maps to obtain a heat map;
[0047] S4. Perform target detection on the heat map through the center point estimation detection algorithm to obtain an output image containing the bounding box of the target (obstacle), its category and confidence.
[0048] The specific method of performing feature processing on the preprocessed image through the gradient reparameterization network to obtain a multi-scale feature map is:
[0049] Construct a gradient reparameterized network for feature processing: perform hyperparameter search on the preset data set, obtain the scale value for optimizing the gradient and extract the constant multiplier from the scale value; create an optimizer for training, multiply the original gradient by the constant multiplier to obtain the optimized gradient, update the optimizer with the optimized gradient, introduce the VGG model that only contains a 3x3 convolution operator, and combine the VGG model and the updated optimizer to obtain the gradient reparameterized network;
[0050] The gradient reparameterization network is used to perform forward propagation, backward propagation, and parameter update on the preprocessed image to obtain a multi-scale feature map.
[0051] The specific method of fusing the multi-scale feature map to obtain the heat map is as follows: increasing the network's receptive field through spatial pyramid pooling SPP and separating multiple features of the multi-scale feature map; fusing the multi-scale feature map through the feature pyramid network FPN and the path aggregation network PAN, and outputting the heat map.
[0052] The method of performing target detection on the heat map through the center point estimation detection algorithm to obtain the output image containing the bounding box of the target (obstacle), its category and confidence is as follows:
[0053] Perform three upsampling operations on the heat map to reduce the resolution of its feature map;
[0054] Add three head structures to the heat map after the resolution is reduced, which are used to predict the center point heat map, center point coordinate compensation and bounding box size respectively, and determine the category of the heat map target according to the predicted center point heat map, center point coordinate compensation and bounding box size;
[0055] Predict the center point and confidence of each target of each category based on the center point heat map. The center point of the target is the peak of the heat map, and regress the center point coordinate compensation and the width and height of the bounding box.
[0056] Assume that the coordinates of the center point of a target are (x i ,y i ), coordinate compensation is (δx i ,δy i ), width and height are (w i ,h i ), then the coordinates of the upper left corner and the lower right corner of the target bounding box are represented as:
[0057] Coordinates of the upper left corner: (x i +δx i -w i / 2,y i +δy i -h i / 2)
[0058] Coordinates of the lower right corner: (x i +δx i +w i / 2,y i +δy i +h i / 2)
[0059] The bounding box of the target is determined based on the coordinates of the upper left corner and the lower right corner.
[0060] Preprocessing includes at least color space enhancement, image size conversion and pixel normalization.
[0061] Color space enhancement includes at least contrast, saturation, hue, and blur.
[0062] Image resizing is a scaling strategy that keeps the aspect ratio unchanged. When the input image does not meet the required aspect ratio, the low-information area is removed, the bottom of the image is cropped, and the cropped image is scaled to the required size using the nearest neighbor or bilinear sampling method.
[0063] Pixel normalization is used to make the input image data have the same distribution. Z-score normalization is used, that is, the distribution with a mean of 0 and a standard deviation of 1. The formula is:
[0064]
[0065] Among them, z is the image data after pixel normalization, x is the acquired image, μ is the mean of the data in the preset data set, and σ is the standard deviation of the data in the preset data set.
[0066] Embodiment 2:
[0067] A target detection system for autonomous driving based on gradient reparameterization and center point estimation is also provided, comprising: a preprocessing module, a feature extraction module, a feature fusion module and a target detection module; wherein,
[0068] The preprocessing module is used to preprocess the images collected during autonomous driving. The embodiment of the present invention uses the fisheye images collected by the on-board surround fisheye camera as input data, and performs preprocessing such as color space enhancement, image size conversion, and pixel normalization. Among them, color space enhancement includes operations such as contrast, saturation, hue, and blurring. The image size is scaled to keep the aspect ratio unchanged. When the input image does not meet the required aspect ratio, the area with low information content is removed, usually the bottom of the cropped image, and then the cropped image is scaled to the required size of the model using the nearest neighbor or bilinear sampling method. The present invention uses 640×480. Pixel normalization makes the input data have the same distribution, which can accelerate model convergence and increase generalization ability. The present invention uses z-score normalization, that is, a distribution with a mean of 0 and a standard deviation of 1. The formula is:
[0069] A feature extraction module is used to perform feature processing on the preprocessed image through a gradient reparameterization network to obtain a multi-scale feature map; the module is used to extract abstract, high-level semantic features embedded in the image. The embodiment of the present invention uses a novel, simple and efficient gradient reparameterization network, and the steps are as follows: 1) HS is performed based on the search data set (CIFAR-100) to obtain the scale value (scales) used to optimize the gradient, and a constant multiplier (Grad Mult, GradM) is extracted from the scales; 2) an optimizer for training is created, such as SGD, and the GradM in 1) is multiplied with the original gradient to obtain the optimized gradient for updating the optimizer; 3) a conventional VGG model is created, that is, only a 3x3 convolution operator is included; 4) forward propagation, backward propagation and parameter update of the network are performed on the target data set according to the VGG model and the SGD optimizer. It is worth noting that scales only depends on the model structure and has nothing to do with the search data set. The search data set in step 1) can also be other data sets, such as ImageNet. In actual operation, you only need to download the open source scales weight file and load it into the code environment. No additional HS operation is required. After specifying the optimizer and the corresponding model, you can train on the target dataset. For an input of 640×480, Backbone will generate feature maps of three sizes: 80×60, 40×30, and 20×15 for the feature fusion module.
[0070] The feature fusion module is used to fuse the multi-scale feature map to obtain a heat map; this module is used to fuse features of different scales. The embodiment of the present invention mainly uses three mechanisms: SPP, FPN and PAN. First, SPP is added to the 20×15 feature map output by the top layer of Backbone. Here, the maximum pooling of four scales of 1×1, 3×3, 5×5 and 7×7 is used to increase the receptive field of the network and separate context features; secondly, FPN is performed to upsample the high-level, low-resolution 40×30 and 20×15 feature maps and pass them downward, and fuse them with the low-level, high-resolution 80×60 feature maps to enhance the network's semantic information representation capabilities; finally, PAN is applied to downsample the low-level, high-resolution 80×60 and 40×30 information representations and pass them upward, and fuse them with the high-level, low-resolution 20×15 features to enhance the network's position information representation capabilities. The heat map output at the end contains rich semantic information, position information and global features.
[0071] The target detection module is used to perform target detection on the heat map through the center point estimation detection algorithm to obtain an output image containing the bounding box of the target (obstacle), the category to which it belongs, and the confidence. The embodiment of the present invention adopts a key point (center point) based detection algorithm that does not require an anchor frame and does not require NMS post-processing. The specific approach is: first, based on the enhanced heat map output by Neck, three upsampling operations are performed to change the feature map resolution to 160×120; secondly, three head structures are added to predict the center point heat map Y∈[0,1] respectively. 160×120×C , center point coordinate compensation O∈R 160×120×2 and the bounding box size S∈R 160×120×2 . Where C is a predefined target category, such as pedestrians, animals, motor vehicles, cones, etc. The center point of each category of target is predicted based on the center point heat map, that is, the peak of the heat map, and the center point coordinate compensation and the width and height of the bounding box are regressed. Assume that the center point coordinates of a certain target are (x i ,y i ), coordinate compensation is (δx i ,δy i ), width and height are (w i ,h i ), then the coordinates of the upper left and lower right corners of the target bounding box can be represented as:
[0072] Upper left: (x i +δx i -w i / 2,y i +δy i -h i / 2)
[0073] Bottom right: (xi +δx i +w i / 2,y i +δy i +h i / 2)
[0074] Based on the above technical route, the target detection function of the original fisheye image is realized, and the output image contains the bounding box of the obstacle, its category and confidence, thereby providing perception information of the surrounding environment for autonomous driving planning decisions.
[0075] The workflow diagram of the system is as follows: Figure 2 As shown, HS (Hyper-Search): hyper-parameter search; NMS (Non-Maximum Suppression): non-maximum suppression; SGD (Stochastic Gradient Descent): stochastic gradient descent; SPP (Spatial Pyramid Pooling): spatial pyramid pooling; FPN (Feature Pyramid Network): feature pyramid network; PAN (Path Aggregation Network): path aggregation network.
[0076] Embodiment 3:
[0077] The present invention also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0078] A computer program is stored in the memory, and when the program is executed by the processor, the processor is caused to perform the steps of the method.
[0079] Embodiment 4:
[0080] The present invention also provides a computer-readable storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor implements the method described above.
[0081] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0082] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0085] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An autonomous driving target detection method based on gradient reparameterization and center point estimation, characterized in that: include: In autonomous driving, the collected images are preprocessed; The preprocessed image is processed by gradient reparameterization network to obtain a multi-scale feature map; Perform feature fusion on multi-scale feature maps to obtain a heat map; The heat map is used to detect targets through the center point estimation detection algorithm to obtain an output image containing the target's bounding box, category, and confidence level.
2. The automatic driving target detection method based on gradient reparameterization and center point estimation according to claim 1, characterized in that: The specific method of performing feature processing on the preprocessed image through the gradient reparameterization network to obtain a multi-scale feature map is: Construct a gradient reparameterized network for feature processing: perform hyperparameter search on the preset data set, obtain the scale value for optimizing the gradient and extract the constant multiplier from the scale value; create an optimizer for training, multiply the original gradient by the constant multiplier to obtain the optimized gradient, update the optimizer with the optimized gradient, introduce the VGG model that only contains a 3x3 convolution operator, and combine the VGG model and the updated optimizer to obtain the gradient reparameterized network; The gradient reparameterization network is used to perform forward propagation, backward propagation, and parameter update on the preprocessed image to obtain a multi-scale feature map.
3. The automatic driving target detection method based on gradient reparameterization and center point estimation according to claim 1, characterized in that: The specific method of fusing the multi-scale feature map to obtain the heat map is as follows: increasing the network's receptive field through spatial pyramid pooling SPP and separating multiple features of the multi-scale feature map; fusing the multi-scale feature map through the feature pyramid network FPN and the path aggregation network PAN, and outputting the heat map.
4. The automatic driving target detection method based on gradient reparameterization and center point estimation according to claim 1, characterized in that: The method of performing target detection on the heat map through the center point estimation detection algorithm to obtain the output image containing the obstacle's bounding box, category and confidence is as follows: Perform three upsampling operations on the heat map to reduce the resolution of its feature map; Add three head structures to the heat map after the resolution is reduced, which are used to predict the center point heat map, center point coordinate compensation and bounding box size respectively, and determine the category of the heat map target according to the predicted center point heat map, center point coordinate compensation and bounding box size; Predict the center point and confidence of each target of each category based on the center point heat map. The center point of the target is the peak of the heat map, and regress the center point coordinate compensation and the width and height of the bounding box. Assume that the coordinates of the center point of a target are (x i ,y i ), coordinate compensation is (δx i ,δy i ), width and height are (w i ,h i ), then the coordinates of the upper left corner and the lower right corner of the target bounding box are represented as: Coordinates of the upper left corner: (x i +δx i -w i / 2,y i +δy i -h i / 2) Coordinates of the lower right corner: (x i +δx i +w i / 2,y i +δy i +h i / 2) The bounding box of the target is determined based on the coordinates of the upper left corner and the lower right corner.
5. The automatic driving target detection method based on gradient reparameterization and center point estimation according to claim 1, characterized in that: Preprocessing includes at least color space enhancement, image size conversion and pixel normalization.
6. The automatic driving target detection method based on gradient reparameterization and center point estimation according to claim 5, characterized in that: Color space enhancement includes at least contrast, saturation, hue, and blur.
7. The automatic driving target detection method based on gradient reparameterization and center point estimation according to claim 5, characterized in that: Image resizing is a scaling strategy that keeps the aspect ratio unchanged. When the input image does not meet the required aspect ratio, the low-information area is removed, the bottom of the image is cropped, and the cropped image is scaled to the required size using the nearest neighbor or bilinear sampling method.
8. The automatic driving target detection method based on gradient reparameterization and center point estimation according to claim 5, characterized in that: Pixel normalization is used to make the input image data have the same distribution. Z-score normalization is used, that is, the distribution with a mean of 0 and a standard deviation of 1. The formula is: Among them, z is the image data after pixel normalization, x is the acquired image, μ is the mean of the data in the preset data set, and σ is the standard deviation of the data in the preset data set.
9. Autonomous driving target detection system based on gradient reparameterization and center point estimation, characterized in that: include: Preprocessing module, feature extraction module, feature fusion module and target detection module; among them, A preprocessing module, used to preprocess images collected during autonomous driving; A feature extraction module is used to perform feature processing on the preprocessed image through a gradient reparameterization network to obtain a multi-scale feature map; Feature fusion module, used to fuse multi-scale feature maps to obtain heat maps; The target detection module is used to perform target detection on the heat map through the center point estimation detection algorithm to obtain an output image containing the target's bounding box, category and confidence.
10. The automatic driving target detection system based on gradient reparameterization and center point estimation according to claim 9, characterized in that: The fisheye image acquired by the vehicle-mounted fisheye camera is used as the image of the input preprocessing module.