Visual perception method and device, electronic equipment and storage medium

Abstract

This invention provides a visual perception method, apparatus, electronic device, and storage medium. The method includes: acquiring a target image captured by an onboard camera, including obstacles and / or lane lines; performing a transformation process on the target image to reduce the network input size of the target image, wherein the transformation process is a process of reducing the network input size of the image; inputting the transformed target image into a perception network model; and calculating the structured information of obstacles and / or lane lines in the target image based on the output of the perception network model, wherein the structured information includes obstacle boxes and ground points, and lane line point sets. This invention has a smaller computational load and lower time consumption, which can reduce the hardware requirements of monocular vision ADAS systems, enabling monocular vision ADAS systems with this perception network model to have better stability and more accurate perception capabilities when deployed on low-end devices.

Description

Technical Field

[0001] This invention relates to the field of driver assistance systems, and more particularly to a visual perception method, device, electronic device, and storage medium. Background Technology

[0002] Advanced Driver Assistance Systems (ADAS) utilize various sensors (cameras, navigation systems, and radar, etc.) installed in vehicles to collect environmental data inside and outside the vehicle in a timely manner. They perform technical processing such as identification, detection, and tracking of static and dynamic objects, enabling drivers to detect potential safety hazards in the shortest possible time and thus improve driving safety.

[0003] As the core of a monocular vision ADAS system, 2D visual perception accounts for almost all of the system's computational load. The remaining computational load is mainly consumed in camera data stream processing, FCW (Forward Collision Warning) and LDW (Lane Departure Warning) logic processing. Deploying a monocular vision ADAS system on low-end devices (such as the K5-T car dashcam, which only has a built-in 6-core ARMv7 CPU@1GHz without an acceleration chip, or even the even less powerful 03T device) and achieving a relatively high frame rate (e.g., 10FPS, or 100ms per frame) is extremely difficult. At frame rates that are too low (e.g., 5FPS), due to the difficulty in achieving uniform time interval sampling and the poor performance of the perceptual neural network on low-end devices, significant deviations in the perception results can occur. Furthermore, on low-end devices (which may even lack a heatsink), the continuous high computational load can overload device resources and maintain extremely high temperatures, posing a greater challenge to system performance stability.

[0004] In fact, reducing the processing time of a single monocular 2D obstacle detection (bounding boxes and grounding points, etc.) to 100ms per frame while maintaining relatively normal detection performance on low-end devices is extremely difficult. At least, no publicly available research or open-source solution has been provided for this. Basic monocular ADAS 2D perception involves at least two tasks: obstacle and lane line perception, totaling at least 200ms (processing 5 frames per second), placing an excessive burden on such low-end devices. Therefore, reducing the processing time of a monocular ADAS 2D perception system to 100ms while maintaining relatively normal and usable perception performance is crucial for mass production on low-end devices. Summary of the Invention

[0005] This invention provides a visual perception method, device, electronic device, and storage medium to address the shortcomings of existing monocular vision ADAS systems deployed on low-end devices, which lead to significant deviations in perception results and poor system performance stability. This invention enables accurate perception and stable application of monocular vision ADAS systems on low-end devices.

[0006] This invention provides a visual perception method, comprising:

[0007] Acquire target images captured by the vehicle-mounted camera, including obstacles and / or lane lines;

[0008] The target image is transformed to reduce its network input size; the transformation process is a process to reduce the network input size of the image.

[0009] The transformed target image is input into a perceptual network model. Based on the output of the perceptual network model, the structured information of obstacles and / or lane lines in the target image is calculated. The structured information includes obstacle boxes and ground points, as well as lane line point sets.

[0010] The perception network model is based on a neural network with multiple detection heads. It is trained using training images that include obstacles and / or lane lines, and are labeled with information encoded to obtain a label supervision signal, and have undergone transformation processing. The label information includes obstacle boxes, grounding points, and lane line point sets. The label supervision signal includes obstacle and lane line positioning heatmaps, quantization deviation between obstacle center point and lane line sampling point, obstacle width and height, offset from obstacle center point to grounding point, and offset from lane line sampling point to starting point. The neural network is used to extract and fuse features from the transformed training images to obtain feature maps, and is used to perform multi-head detection on the feature maps. During detection, the obstacle and lane line positioning heatmaps share a first detection head, and the quantization deviation regression between obstacle center point and lane line sampling point shares a second detection head.

[0011] According to a visual perception method provided by the invention, the neural network is further used to regress the width and height of the obstacle and the offset from the center point of the obstacle to the ground point in the feature map using a third detection head and a fourth detection head, respectively.

[0012] According to a visual perception method provided by the invention, the step of calculating the structured information of obstacles in a target image based on the output of the perception network model includes:

[0013] The obstacle frame and grounding point are calculated based on the obstacle positioning heatmap output by the first detection head, the obstacle center point quantization deviation output by the second detection head, the obstacle width and height output by the third detection head, and the obstacle center point offset to the grounding point output by the fourth detection head at the size of the feature map.

[0014] An affine transformation is performed on the obstacle bounding box and ground point at the feature map size to obtain the obstacle bounding box and ground point at the target image size.

[0015] According to a visual perception method provided by the invention, the neural network is also used to regress the offset from the starting point to the lane line sampling point in the feature map using a fifth detection head.

[0016] According to a visual perception method provided by the invention, the step of calculating the structured information of lane lines in a target image based on the output of the perception network model includes:

[0017] The actual lane line sampling point and starting point at the specified feature map size are calculated based on the lane line positioning heatmap output by the first detection head, the lane line sampling point quantization deviation output by the second detection head, and the lane line sampling point offset from the starting point output by the fifth detection head.

[0018] An affine transformation is performed on the positions of the actual lane line sampling points and the starting point at the feature map size to obtain the lane line point set at the target image size.

[0019] According to a visual perception method provided by the invention, the neural network is a network formed by extending GANet to the CenterNet framework. Feature extraction and fusion are performed on the transformed training image to obtain a feature map, and multi-head detection is performed on the feature map, including:

[0020] A lightweight backbone network and Neck are used to sequentially perform feature extraction and feature fusion enhancement on the transformed training image to obtain the feature map.

[0021] The obstacles and lane lines are detected using CenterNet and GANet respectively.

[0022] According to the visual perception method provided by the invention, a perception network model is obtained by training a training image containing labeled supervision signals (including obstacles and / or lane lines) encoded with labeled information and after transformation processing, based on a neural network with multiple detection heads. The model includes:

[0023] Loss calculation is performed on the outputs of each detection head and the label supervision signal transformed from the training image size to the feature map size;

[0024] The gradient is backpropagated based on the loss calculation results to iterate and improve the constructed perceptual network model.

[0025] The present invention also provides a visual sensing device, the device comprising:

[0026] The acquisition module is used to acquire target images, including obstacles and / or lane lines, captured by the vehicle-mounted camera;

[0027] A transformation module is used to perform transformation processing on the target image to reduce the network input size of the target image, wherein the transformation processing is a process of reducing the network input size of the image;

[0028] The calculation module inputs the transformed target image into a perceptual network model, and calculates the structured information of obstacles and / or lane lines in the target image based on the output of the perceptual network model. The structured information includes obstacle boxes and ground points, as well as lane line point sets.

[0029] The perception network model is based on a neural network with multiple detection heads. It is trained using training images that include obstacles and / or lane lines, and are labeled with information encoded to obtain a label supervision signal, and have undergone transformation processing. The label information includes obstacle boxes, grounding points, and lane line point sets. The label supervision signal includes obstacle and lane line positioning heatmaps, quantization deviation between obstacle center point and lane line sampling point, obstacle width and height, offset from obstacle center point to grounding point, and offset from lane line sampling point to starting point. The neural network is used to extract and fuse features from the transformed training images to obtain feature maps, and is used to perform multi-head detection on the feature maps. During detection, the obstacle and lane line positioning heatmaps share a first detection head, and the quantization deviation regression between obstacle center point and lane line sampling point shares a second detection head.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the visual perception method as described above.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the visual perception method as described above.

[0032] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the visual perception method as described above.

[0033] The visual perception method, device, electronic device, and storage medium provided by this invention train a neural network using training images. During training, the training images are reduced in size after transformation and the detection heads are merged to reduce computational load and time consumption. This results in the training and construction of a perception network model with lower computational load and time consumption. Because of its lower computational load and time consumption, this perception network model can reduce the hardware requirements of a monocular vision ADAS system, enabling a monocular vision ADAS system with this perception network model to have better stability and more accurate perception capabilities when deployed on low-end devices. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is one of the flowcharts of the visual perception method provided by the present invention;

[0036] Figure 2 This is the second flowchart of the visual perception method provided by the present invention;

[0037] Figure 3 This is the third flowchart of the visual perception method provided by the present invention;

[0038] Figure 4 This is the fourth flowchart of the visual perception method provided by the present invention;

[0039] Figure 5 This is the fifth flowchart of the visual perception method provided by the present invention;

[0040] Figure 6 This is the sixth flowchart of the visual perception method provided by the present invention;

[0041] Figure 7 This is a diagram of the target detection network architecture provided by the present invention;

[0042] Figure 8 This is a detection effect diagram of the visual perception method provided by the present invention;

[0043] Figure 9 This is a schematic diagram of the structure of the visual perception device provided by the present invention;

[0044] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0045] Figure label:

[0046] 910: Acquisition module; 920: Transformation module; 930: Calculation module; 1010: Processor; 1020: Communication interface; 1030: Memory; 1040: Communication bus. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0048] The following is combined Figures 1-10 The present invention describes a visual perception method, apparatus, electronic device, and storage medium.

[0049] like Figure 1 As shown, in one embodiment, a visual perception method includes the following steps:

[0050] Step S110: Acquire a target image captured by the vehicle-mounted camera, including obstacles and / or lane lines.

[0051] Specifically, in actual use, it is necessary to acquire real-time images of the target on site through vehicle-mounted cameras, and these images must include obstacles and / or lane lines in order to achieve target perception.

[0052] Step S120: Transform the target image to reduce the network input size of the target image. The transformation process is a process to reduce the network input size of the image.

[0053] Preferably, the image size is transformed to a network input size of 144×256 after preprocessing, which reduces the resolution and computational load of the network compared to the traditional network input size of 512×512.

[0054] Step S130: Input the transformed target image into the perceptual network model, and calculate the structured information of obstacles and / or lane lines in the target image based on the output of the perceptual network model. The structured information includes obstacle boxes and grounding points, as well as lane line point sets.

[0055] The perception network model is based on a neural network with multiple detection heads. It is trained using training images that include labeled supervision signals (coded with labeled information, including obstacles and / or lane lines) and have undergone transformation processing. The labeled information includes obstacle bounding boxes, grounding points, and lane line point sets. The labeled supervision signals include obstacle and lane line localization heatmaps, quantization deviation between the obstacle center point and lane line sampling points, obstacle width and height, offset from the obstacle center point to the grounding point, and offset from the lane line sampling point to the starting point. The neural network is used to extract and fuse features from the transformed training images to obtain feature maps, which are then used for multi-head detection. During detection, the obstacle and lane line localization heatmaps share a first detection head, while the quantization deviation regression between the obstacle center point and lane line sampling points shares a second detection head.

[0056] Among them, the obstacle and lane line positioning heatmap is obtained by positioning the center point of the obstacle frame and the sampling point of the lane line.

[0057] Specifically, training images are selected and labeled according to the detection target of the perceptual network model. The selected training images should contain obstacles and / or lane lines, and the obstacles and lane lines in the selected training images need to be labeled to form training images with label information that can meet the training objectives. In addition, in order to facilitate recognition by the neural network, the label information also needs to be encoded to obtain label supervision signals for the neural network to recognize.

[0058] It should be noted that in order for the trained perception network model to meet the detection requirements in real driving scenarios, it is necessary to collect training images under different scenarios and label them, such as training images under different weather conditions, daytime, nighttime, different roads, and different traffic flows.

[0059] Since the perceptual network model built after training is designed for low-end devices, the computational load and time during training can be reduced by shrinking the training images and reducing the number of detection heads, so that the built perceptual network model can be used on low-end devices.

[0060] Specifically, after the neural network extracts and fuses features from the transformed training image to obtain a feature map, the detection heads used for obstacle detection heatmap localization and lane line detection heatmap localization are merged, i.e., the first detection head is used for detection. Simultaneously, the detection heads used for center point quantization deviation and sampling point quantization deviation are merged, i.e., the second detection head is used for detection. The merged detection heads reduce the computational load and time consumption of the network, thereby lowering the computational requirements of the constructed perceptual network model.

[0061] Since the training is performed using transformed training images, the information about obstacles and lane lines output by the perceptual network model when perceiving the target image is based on the transformed target image size. Therefore, calculations are required to obtain the structured information of obstacles and lane lines at the target image size.

[0062] The visual perception method in this embodiment trains the neural network using training images. During training, the training images are reduced in size after transformation and the detection heads are merged to reduce the amount of computation and time consumption. This results in the training and construction of a perception network model with lower computational and time requirements. Because of its lower computational and time requirements, this perception network model can reduce the hardware requirements of the monocular vision ADAS system. As a result, the monocular vision ADAS system with this perception network model can have better stability and more accurate perception capabilities when deployed on low-end devices.

[0063] like Figure 2 As shown, in one embodiment, the visual perception method of the present invention further includes a neural network used to detect the width and height of obstacles and the offset from the center point of the obstacle to the ground point in the feature map using a third detection head and a fourth detection head, respectively.

[0064] Specifically, during vehicle operation, to obtain accurate structured information about obstacles, in addition to determining the quantization deviation between the obstacle's center point and the center point, it is also necessary to acquire the obstacle's width and height, as well as the offset from the obstacle's center point to the ground point. Therefore, during training, the label information of the training images needs to include the obstacle's width and height, as well as the offset from the obstacle's center point to the ground point. Furthermore, during training, a third detection head and a fourth detection head are used to detect the obstacle's width and height, as well as the offset from the obstacle's center point to the ground point, in the feature images, respectively.

[0065] In this embodiment, the structured information of obstacles in the target image is calculated based on the output of the perception network model, including the following steps:

[0066] Step S122: Calculate the obstacle frame and grounding point in the feature map size based on the obstacle positioning heatmap output by the first detection head, the obstacle center point quantization deviation output by the second detection head, the obstacle width and height output by the third detection head, and the obstacle center point offset to the grounding point output by the fourth detection head.

[0067] Specifically, after the vehicle-mounted camera captures an image of the target at the scene, the image is transformed. The perceptual network model then performs perception and recognition on the transformed image. The first detection head outputs an obstacle localization heatmap, which can then be converted into the obstacle's center point. The second detection head outputs the quantized deviation of the obstacle's center point, the third detection head outputs the obstacle's width and height, and the fourth detection head outputs the offset from the obstacle's center point to the ground point. These parameters are then used to calculate the obstacle bounding box and the ground point at the feature map size using the following formula:

[0068] Obstacle frame = center point + quantization deviation + width and height

[0069] Grounding point = center point + quantization deviation + offset from center point to grounding point.

[0070] Step S124: Perform an affine transformation on the obstacle box and ground point when the feature map size is used to obtain the obstacle box and ground point when the target image size is used.

[0071] Specifically, the obstacle bounding boxes and grounding points calculated above are at the size of the feature map. Therefore, an affine transformation is needed to transform the obstacle bounding boxes and grounding points to the size of the target image.

[0072] like Figure 3 As shown, in one embodiment, the visual perception method of the present invention further includes a neural network used to detect the offset of lane line sampling points from the starting point in the feature map using a fifth detection head.

[0073] Specifically, during vehicle movement, in order to group all sampling points belonging to the same lane line together, in addition to determining the quantization deviation between lane line sampling points, it is also necessary to obtain the offset of the lane line sampling points from the starting point. Therefore, during training, a fifth detector is used to detect the offset of lane line sampling points from the starting point in the feature map.

[0074] In this embodiment, the structured information of lane lines in the target image is calculated based on the output of the perception network model, including the following steps:

[0075] Step S123: Calculate the actual lane line sampling point and starting point at the feature map size based on the lane line positioning heatmap output by the first detection head, the sampling point quantization deviation output by the second detection head, and the lane line sampling point offset to the starting point output by the fifth detection head.

[0076] Specifically, after the vehicle-mounted camera captures the target image, the perceptual network model performs perception and recognition on the transformed target image. The first detection head outputs a lane line localization heatmap, which can then be converted into lane line sampling point localization. The second detection head outputs the sampling point quantization deviation, and the fifth detection head outputs the lane line sampling point offset from the starting point. The actual lane line sampling points and starting points are calculated using the following formula at the feature image size:

[0077] Actual sampling points = sampling points + quantization bias

[0078] Starting point = sampling point + quantization deviation + offset of sampling point from starting point.

[0079] Step S125: Perform an affine transformation on the actual lane line sampling points and starting points when the feature map size is used to obtain the actual lane line sampling points and starting points when the target image size is used.

[0080] Specifically, the lane line's true sampling points and starting points calculated above are at the size of the feature map. Therefore, an affine transformation is needed to transform the lane line's true sampling points and starting points to the size of the target image.

[0081] like Figure 4 As shown, in one embodiment, the visual perception method of the present invention uses a neural network formed by extending GANet onto the CenterNet framework.

[0082] Specifically, based on the bottom-up concept and using the CenterNet framework as a foundation, GANet (Global Association Network) is extended onto the CenterNet framework to build a neural network that integrates obstacle perception network and lane line perception network.

[0083] Feature extraction and fusion are performed on the transformed training images to obtain feature maps, and multi-head detection is then performed on the feature maps, including the following steps:

[0084] In step S410, a lightweight backbone network and Neck are used to sequentially perform feature extraction and feature fusion enhancement on the transformed training image to obtain a feature image.

[0085] Among them, using CenterNet for obstacle detection and GANet for lane detection are both feasible and can be simplified to low-end devices. Secondly, both are based on the bottom-up idea, that is, they both obtain feature images with a certain downsampling stride relative to the input size through the backbone network and Neck.

[0086] Step S420: Detect obstacles and lane lines based on CenterNet and GANet respectively.

[0087] In CenterNet, four heads (detection heads) slide on the feature map for obstacle center point localization, center point quantization deviation regression, obstacle width and height regression, and center point to ground point offset regression, respectively. Similarly, three heads in GANet slide on the feature map for lane line sampling point localization, sampling point quantization deviation regression, and sampling point to starting point offset regression, respectively. Therefore, the object detection network is essentially a backbone network + Neck + seven heads. Since the Neck output feature map size is H / r × W / r (where H and W are the input height and width, respectively, and r is the downsampling stride of the feature image), and the heads slide on the feature map, the corresponding computational cost is head computation cost × H / r × W / r, which is a key computational bottleneck. To address this computational bottleneck, the obstacle center point localization head and the lane line sampling point localization head are further merged (the number of output channels is simply the number of obstacle categories + 1, where 1 represents the lane line category). The obstacle center point quantization deviation regression head and the lane line sampling point quantization deviation regression head are also merged (the number of output channels remains unchanged, but is shared by both obstacles and lane lines). This reduces the computation time of the two heads, resulting in a neural network with only one more head of computation than the obstacle perception network. Therefore, the latency of the entire monocular ADAS 2D perception system can be reduced to almost the same as that of the obstacle perception network. Furthermore, both the obstacle perception network based on the CenterNet framework and the lane line perception network based on the GANet framework can achieve a frame latency of 100ms with relatively normal detection performance on low-end devices after scaling down. The perception network model built from the merged object detection network, after appropriate training, shows little difference in performance compared to the individual obstacle perception network and lane line perception network.

[0088] like Figure 5 As shown, in one embodiment, a perceptual network model is trained based on a neural network with multiple detection heads, using training images that include labeled supervision signals obtained by encoding labeled information, including obstacles and / or lane lines, and after transformation processing, to obtain a perceptual network model, including the following steps:

[0089] Step S510: Calculate the loss of the output of each detection head and the label supervision signal transformed from the training image size to the feature map size.

[0090] Step S520: Backpropagate gradients based on the Loss calculation results to iterate and improve the constructed perceptual network model.

[0091] Backpropagation is the specific implementation of gradient descent in neural networks. Gradient descent and backpropagation have greatly advanced the fields of machine learning and deep learning, enabling better and more convenient parameter tuning to obtain better perceptual network models.

[0092] like Figure 6 As shown in the specific embodiment, due to the single task of monocular 2D obstacle detection (boundary boxes and grounding points, etc.), it is extremely difficult to reduce the network latency to 100ms per frame while maintaining relatively normal detection performance on low-end devices. At least publicly available research and open-source solutions have not provided a solution for this situation. Monocular vision ADAS 2D perception includes at least two tasks: obstacle and lane line perception, totaling at least 200ms (processing 5 frames per second), which places an excessive burden on such low-end devices. Therefore, reducing the latency of the monocular vision ADAS 2D perception system to 100ms while maintaining relatively normal and usable perception performance is crucial for mass production on low-end devices. The visual perception method of this invention, based on the bottleneck concept, constructs an end-to-end unified 2D obstacle and lane line perception network model, which can reduce the latency of the entire monocular vision ADAS perception system to 100ms on low-end devices, enabling the entire monocular ADAS system to achieve over 10 FPS on low-end devices.

[0093] In this embodiment, the visual perception method of the present invention includes the following steps:

[0094] Step S610: Sample training images that meet the expected distribution based on actual driving scenarios to form an image dataset.

[0095] The scenarios include driving scenarios such as roads, traffic flow, weather (sunny, rainy, foggy, snowy, etc.), and day and night conditions. A training image dataset containing obstacles and lane lines with labeled information is created based on the expected output annotations of the training images. The obstacles are derived from 17 common obstacle categories: high-speed scene obstacles, non-high-speed scene obstacles, and pedestrians. Each obstacle has a maximum of four grounding points.

[0096] Step S620: Based on the CenterNet framework, GANet is extended to form a neural network on the CenterNet framework, and training images are used to train and build a perceptual network model.

[0097] Since the constructed perception network model is designed for low-end devices, two key points need to be considered during training. First, the model is extremely lightweight, requiring approximately 18 MFLOPs (Million Floating-point Operations per Second) of computation and about 0.045M parameters. Second, multi-task optimization involves reducing the number of detection heads to five, while still employing seven loss functions to address different tasks, categorized into obstacle and lane line detection. Through these two targeted optimizations, the performance metrics are only slightly different compared to standalone obstacle and lane line perception networks.

[0098] For specific details on the object detection network architecture, please refer to [link / reference]. Figure 7 As shown, the training images are preprocessed and transformed to a network input size of 144×256, then input into a modified lightweight ShuffleNet v2 backbone network (computational cost of only 4.36 MFLOPs), outputting a five-stage feature map. The feature maps from the last four stages are then input into the Neck for multi-scale fusion enhancement and feature integration to obtain the final feature maps (size 144 / 4×256 / 4) used for each task. The neural network has a total of five detection heads: an obstacle and lane line localization detection head, an obstacle and lane line localization quantization deviation detection head, an obstacle width and height detection head, an obstacle center point to ground point 2D offset detection head, and a lane line sampling point to corresponding starting point 2D offset detection head. This step is the most computationally intensive because each detection head needs to slide on a feature image with a stride of 4, meaning the computational cost of the detection head needs to be multiplied by the feature image resolution. The neural network completes the obstacle and lane line perception tasks with only a minimal increase in computational cost compared to a single obstacle perception network with one more head. During training, the outputs of each detection head and the supervision signal transformed from the training image to the feature map size are used to calculate the loss, and the gradient is backpropagated for iteration.

[0099] Step S630: Capture target images of the vehicle's driving scene using an onboard camera, input the target images into the constructed perception network model, and calculate the structured information of obstacles and lane lines based on the output of the perception network model.

[0100] Specifically, the first three channels of the location heatmap output by the obstacle and lane line localization detection head are used as the obstacle heatmap (corresponding to the three obstacle categories respectively), and the fourth channel is used as the lane line heatmap, and both are decoded. For the obstacle heatmap, the 3×3 local maximum point with confidence belonging to the top K is selected as the center point of the detected obstacle. Then, based on the position index, the quantization deviation, width and height, and center point to ground point 2D offset prediction of the corresponding detected target are extracted from the quantization deviation task head output of the obstacle and lane line task head, the obstacle width and height task head output, and the center point to ground point 2D offset task head output, respectively. The bounding box and ground point are obtained by the following formula:

[0101] Obstacle frame = center point + quantization deviation + width and height

[0102] Grounding point = center point + quantization deviation + 2D offset from center point to grounding point.

[0103] However, the bounding box and grounding point are located at the size of the tag supervision signal, i.e., 144 / 4 × 256 / 4. An affine transformation is needed to transform the obstacle bounding box and grounding point to the original image size. Before obtaining the final result, thresholding and NMS (non-maximum suppression) are sometimes required.

[0104] For the lane line heatmap, the 1×3 local maximum points with confidence greater than the threshold are selected as the detected lane line sampling points. Then, based on the location index, the quantization deviation of the corresponding detection sampling point and the 2D offset from the sampling point to the starting point are predicted from the obstacle and lane line quantization deviation task head output and the sampling point to starting point 2D offset task head output, respectively. The sampling point and the corresponding estimated starting point are obtained by the following formula:

[0105] Actual sampling points = sampling points + quantization bias

[0106] Starting point = sampling point + quantization deviation + 2D offset from sampling point to starting point.

[0107] In 2D offset, lane starting points with a ordinate offset less than a threshold (set to 1) are considered potential lane start points. After clustering, the true lane start points (cluster centers) are obtained. Each sampling point, plus quantization bias and the 2D offset to the starting point, yields the estimated lane start point from that sampling point. Lanes whose distance to the cluster center is less than a given threshold belong to that cluster center. However, these lane lines are based on the size of the supervisory signal (144 / 4 × 256 / 4), requiring affine transformation to convert the obstacle boxes and grounding points to the original image size.

[0108] The visual perception method in this embodiment employs an end-to-end unified 2D obstacle and lane line perception network model for monocular vision ADAS. It can reduce the latency of the entire monocular ADAS perception system to 100ms on low-end devices (such as the K5-T car dashcam, which only has a built-in 6-core ARMv7 CPU@1GHz without an acceleration chip, or even the 03T device with its even lower computing power, whose computing chip costs only tens of dollars). This allows the entire monocular ADAS system to achieve over 10 FPS. This avoids a series of problems such as excessive load and frame drops leading to reduced tracking accuracy during actual deployment, providing key technical support for the mass production of monocular ADAS on low-end devices. Trained on a large dataset collected from real-world driving scenarios, this algorithm also shows good adaptability in darkness and extreme weather (rain, fog, snow). Actual detection results are as follows: Figure 8 As shown.

[0109] The visual sensing device provided by the present invention will be described below. The visual sensing device described below can be referred to in correspondence with the visual sensing method described above.

[0110] like Figure 9 As shown, in one embodiment, the visual sensing device of the present invention includes an acquisition module 910, a transformation module 920, and a calculation module 930.

[0111] The acquisition module 910 is used to acquire target images, including obstacles and / or lane lines, captured by the vehicle-mounted camera.

[0112] The transformation module 920 is used to perform transformation processing on the target image to reduce the network input size of the target image. The transformation processing is the process of reducing the network input size of the image.

[0113] The calculation module 920 is used to input the transformed target image into the perception network model, and calculate the structured information of obstacles and / or lane lines in the target image based on the output of the perception network model. The structured information includes obstacle boxes and ground points, as well as lane line point sets.

[0114] The perception network model is based on a neural network with multiple detection heads. It is trained using training images that include obstacles and / or lane lines, and are labeled with information encoded to obtain a label supervision signal, and have undergone transformation processing. The label information includes obstacle boxes, grounding points, and lane line point sets. The label supervision signal includes obstacle and lane line positioning heatmaps, quantization deviation between obstacle center point and lane line sampling point, obstacle width and height, offset from obstacle center point to grounding point, and offset from lane line sampling point to starting point. The neural network is used to extract and fuse features from the transformed training images to obtain feature maps, and is used to perform multi-head detection on the feature maps. During detection, the obstacle and lane line positioning heatmaps share a first detection head, and the quantization deviation regression between obstacle center point and lane line sampling point shares a second detection head.

[0115] In this embodiment, the visual perception device of the present invention, specifically the computing module 930 is used for:

[0116] The obstacle frame and grounding point are calculated based on the obstacle positioning heatmap output by the first detection head, the obstacle center point quantization deviation output by the second detection head, the obstacle width and height output by the third detection head, and the obstacle center point offset to the grounding point output by the fourth detection head at the size of the feature map.

[0117] An affine transformation is performed on the obstacle bounding box and ground point at the feature map size to obtain the obstacle bounding box and ground point at the target image size.

[0118] The neural network is also used to regress the width and height of obstacles and the offset from the center point of the obstacle to the ground point in the feature map using the third and fourth detection heads, respectively.

[0119] In this embodiment, the visual perception device of the present invention, specifically the computing module 930 is used for:

[0120] The actual lane line sampling point and starting point at the specified feature map size are calculated based on the lane line positioning heatmap output by the first detection head, the lane line sampling point quantization deviation output by the second detection head, and the lane line sampling point offset from the starting point output by the fifth detection head.

[0121] Affine transformation is performed on the positions of the actual lane line sampling points and the starting point at the feature map size to obtain the lane line point set at the target image size;

[0122] The neural network is also used to regress the offset of lane line sampling points from the starting point in the feature map using a fifth detection head.

[0123] In this embodiment, the visual perception device of the present invention further includes a processing module, which is used to extract and fuse features from the transformed training image to obtain a feature map, and to perform multi-head detection on the feature map;

[0124] Specifically used for:

[0125] A lightweight backbone network and Neck are used to sequentially perform feature extraction and feature fusion enhancement on the transformed training images to obtain feature images;

[0126] Obstacle detection and lane line detection are performed using CenterNet and GANet respectively;

[0127] Among them, the neural network is a network formed by extending GANet to the CenterNet framework.

[0128] In this embodiment, the visual perception device of the present invention further includes a training module, which is used to train a perception network model based on a neural network with multiple detection heads, using a training image that includes an obstacle and / or lane line, and a label supervision signal obtained by encoding the label information, and which has undergone transformation processing.

[0129] Specifically used for:

[0130] Loss calculation is performed on the outputs of each detection head and the label supervision signal transformed from the training image size to the feature map size;

[0131] The gradient is backpropagated based on the loss calculation results to iterate and improve the constructed perceptual network model.

[0132] The visual perception device in this embodiment trains the target detection network using training images. During training, the training images are reduced in size after transformation and the detection heads are merged to reduce the amount of computation and time consumption. This trains and constructs a perception network model with lower computational and time requirements. Because of its lower computational and time requirements, the perception network model can reduce the hardware requirements of the monocular vision ADAS system. This allows the monocular vision ADAS system with this perception network model to have better stability and more accurate perception capabilities when deployed on low-end devices.

[0133] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a visual perception method, which includes:

[0134] Acquire target images captured by the vehicle-mounted camera, including obstacles and / or lane lines;

[0135] The target image is transformed to reduce its network input size; the transformation process is a process to reduce the network input size of the image.

[0136] The transformed target image is input into a perceptual network model. Based on the output of the perceptual network model, the structured information of obstacles and / or lane lines in the target image is calculated. The structured information includes obstacle boxes and ground points, as well as lane line point sets.

[0137] The perception network model is based on a neural network with multiple detection heads. It is trained using training images that include obstacles and / or lane lines, and are labeled with information encoded to obtain a label supervision signal, and have undergone transformation processing. The label information includes obstacle boxes, grounding points, and lane line point sets. The label supervision signal includes obstacle and lane line positioning heatmaps, quantization deviation between obstacle center point and lane line sampling point, obstacle width and height, offset from obstacle center point to grounding point, and offset from lane line sampling point to starting point. The neural network is used to extract and fuse features from the transformed training images to obtain feature maps, and is used to perform multi-head detection on the feature maps. During detection, the obstacle and lane line positioning heatmaps share a first detection head, and the quantization deviation regression between obstacle center point and lane line sampling point shares a second detection head.

[0138] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the visual perception method provided by the above methods, the method comprising:

[0140] Acquire target images captured by the vehicle-mounted camera, including obstacles and / or lane lines;

[0141] The target image is transformed to reduce its network input size; the transformation process is a process to reduce the network input size of the image.

[0142] The transformed target image is input into a perceptual network model. Based on the output of the perceptual network model, the structured information of obstacles and / or lane lines in the target image is calculated. The structured information includes obstacle boxes and ground points, as well as lane line point sets.

[0143] The perception network model is based on a neural network with multiple detection heads. It is trained using training images that include obstacles and / or lane lines, and are labeled with information encoded to obtain a label supervision signal, and have undergone transformation processing. The label information includes obstacle boxes, grounding points, and lane line point sets. The label supervision signal includes obstacle and lane line positioning heatmaps, quantization deviation between obstacle center point and lane line sampling point, obstacle width and height, offset from obstacle center point to grounding point, and offset from lane line sampling point to starting point. The neural network is used to extract and fuse features from the transformed training images to obtain feature maps, and is used to perform multi-head detection on the feature maps. During detection, the obstacle and lane line positioning heatmaps share a first detection head, and the quantization deviation regression between obstacle center point and lane line sampling point shares a second detection head.

[0144] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the visual perception methods provided by the methods described above, the method comprising:

[0145] Acquire target images captured by the vehicle-mounted camera, including obstacles and / or lane lines;

[0146] The target image is transformed to reduce its network input size; the transformation process is a process to reduce the network input size of the image.

[0147] The transformed target image is input into a perceptual network model. Based on the output of the perceptual network model, the structured information of obstacles and / or lane lines in the target image is calculated. The structured information includes obstacle boxes and ground points, as well as lane line point sets.

[0148] The perception network model is based on a neural network with multiple detection heads. It is trained using training images that include obstacles and / or lane lines, and are labeled with information encoded to obtain a label supervision signal, and have undergone transformation processing. The label information includes obstacle boxes, grounding points, and lane line point sets. The label supervision signal includes obstacle and lane line positioning heatmaps, quantization deviation between obstacle center point and lane line sampling point, obstacle width and height, offset from obstacle center point to grounding point, and offset from lane line sampling point to starting point. The neural network is used to extract and fuse features from the transformed training images to obtain feature maps, and is used to perform multi-head detection on the feature maps. During detection, the obstacle and lane line positioning heatmaps share a first detection head, and the quantization deviation regression between obstacle center point and lane line sampling point shares a second detection head.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of visual perception, characterized by, include: Acquire target images captured by the vehicle-mounted camera, including obstacles and / or lane lines; The target image is transformed to reduce its network input size; the transformation process is a process to reduce the network input size of the image. The transformed target image is input into a perceptual network model. Based on the output of the perceptual network model, the structured information of obstacles and / or lane lines in the target image is calculated. The structured information includes obstacle boxes and ground points, as well as lane line point sets. The perception network model is based on a neural network with multiple detection heads. It is trained using training images that include obstacles and / or lane lines, and are labeled with information encoded to obtain a label supervision signal, and are transformed. The label information includes obstacle boxes, grounding points, and lane line point sets. The label supervision signal includes obstacle and lane line positioning heatmaps, quantization deviation between obstacle center point and lane line sampling point, obstacle width and height, offset from obstacle center point to grounding point, and offset from lane line sampling point to starting point. The neural network is used to extract and fuse features from the transformed training images to obtain feature maps, and to perform multi-head detection on the feature maps. During detection, the obstacle and lane line positioning heatmaps share a first detection head, and the quantization deviation regression between obstacle center point and lane line sampling point shares a second detection head. The first detection head is obtained by merging the detection head used for detecting obstacle heatmap positioning and the detection head used for detecting lane line heatmap positioning, and the second detection head is obtained by merging the detection head used for detecting center point quantization deviation and the detection head used for detecting sampling point quantization deviation.

2. The visual perception method according to claim 1, characterized in that, The neural network is also used to regress the width and height of obstacles and the offset from the center point of the obstacle to the ground point in the feature map using a third detection head and a fourth detection head, respectively.

3. The visual perception method according to claim 2, characterized in that, The step of calculating the structured information of obstacles in the target image based on the output of the perception network model includes: The obstacle frame and grounding point are calculated based on the obstacle positioning heatmap output by the first detection head, the obstacle center point quantization deviation output by the second detection head, the obstacle width and height output by the third detection head, and the obstacle center point offset to the grounding point output by the fourth detection head at the size of the feature map. An affine transformation is performed on the obstacle bounding box and ground point at the feature map size to obtain the obstacle bounding box and ground point at the target image size.

4. The visual perception method according to claim 1, characterized in that, The neural network is also used to regress the offset of lane line sampling points from the starting point in the feature map using a fifth detection head.

5. The visual perception method according to claim 4, characterized in that, The step of calculating the structured information of lane lines in the target image based on the output of the perception network model includes: The actual lane line sampling point and starting point at the specified feature map size are calculated based on the lane line positioning heatmap output by the first detection head, the lane line sampling point quantization deviation output by the second detection head, and the lane line sampling point offset from the starting point output by the fifth detection head. An affine transformation is performed on the positions of the actual lane line sampling points and the starting point at the feature map size to obtain the lane line point set at the target image size.

6. The visual perception method according to any one of claims 1 to 5, characterized in that, The neural network is based on the CenterNet framework, and GANet is extended to the CenterNet framework to form a network. It extracts and fuses features from the transformed training images to obtain feature maps, and performs multi-head detection on the feature maps, including: A lightweight backbone network and Neck are used to sequentially perform feature extraction and feature fusion enhancement on the transformed training image to obtain the feature map. The obstacles and lane lines are detected using CenterNet and GANet respectively.

7. The visual perception method according to any one of claims 1 to 5, characterized in that, Based on a neural network with multiple detection heads, a perceptual network model is obtained by training training images containing labeled supervision signals (encoded with labeled information including obstacles and / or lane lines) and after transformation processing. The model includes: Loss calculation is performed on the outputs of each detection head and the label supervision signal transformed from the training image size to the feature map size; The gradient is backpropagated based on the loss calculation results to iterate and improve the constructed perceptual network model.

8. A visual sensing device, characterized in that, The device includes: The acquisition module is used to acquire target images, including obstacles and / or lane lines, captured by the vehicle-mounted camera; A transformation module is used to perform transformation processing on the target image to reduce the network input size of the target image, wherein the transformation processing is a process of reducing the network input size of the image; The calculation module inputs the transformed target image into a perceptual network model, and calculates the structured information of obstacles and / or lane lines in the target image based on the output of the perceptual network model. The structured information includes obstacle boxes and ground points, as well as lane line point sets. The perception network model is based on a neural network with multiple detection heads. It is trained using training images that include obstacles and / or lane lines, and are labeled with information encoded to obtain a label supervision signal, and are transformed. The label information includes obstacle boxes, grounding points, and lane line point sets. The label supervision signal includes obstacle and lane line positioning heatmaps, quantization deviation between obstacle center point and lane line sampling point, obstacle width and height, offset from obstacle center point to grounding point, and offset from lane line sampling point to starting point. The neural network is used to extract and fuse features from the transformed training images to obtain feature maps, and to perform multi-head detection on the feature maps. During detection, the obstacle and lane line positioning heatmaps share a first detection head, and the quantization deviation regression between obstacle center point and lane line sampling point shares a second detection head. The first detection head is obtained by merging the detection head used for detecting obstacle heatmap positioning and the detection head used for detecting lane line heatmap positioning, and the second detection head is obtained by merging the detection head used for detecting center point quantization deviation and the detection head used for detecting sampling point quantization deviation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the visual perception method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the visual perception method as described in any one of claims 1 to 7.

Images