A blind area monitoring system based on a low-power processing unit of a commercial vehicle platform

Abstract

The present application relates to the technical field of intelligent auxiliary driving of automobiles, and particularly relates to a blind area monitoring system based on a low-computing-power processing unit of a commercial vehicle platform, a blind area range calibration module calibrates a blind area monitoring camera, and provides a preprocessing operation; an image processing module and a perception algorithm module are used for receiving blind area perception information of the camera, judging blind area perception results through an image processing algorithm and a deep learning algorithm, and customizing a blind area monitoring model; an information processing module provides computing power support for the blind area perception model, customizes the blind area monitoring model, and improves inference speed and inference accuracy of the model; a warning prompt module is used for feeding back potential danger warning information to a driver. Through combination of blind area range calibration and customized strategies such as image processing and perception algorithms, an efficient real-time blind area monitoring system based on a low-computing-power information processing unit is realized, which can send warning information to the driver in real time, has low system power consumption, and has strong robustness and scalability.

Description

Technical Field

[0001] This invention relates to the field of intelligent assisted driving technology for automobiles, and specifically to a blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform. Background Technology

[0002] Commercial vehicles typically have large body sizes and blind spots, significantly increasing the difficulty of driver operation and the risk of accidents. Statistics show that blind spot factors account for as much as 35% of truck accidents. Therefore, developing effective blind spot monitoring methods is crucial for the commercial vehicle industry. Traditional blind spot monitoring in commercial vehicles mainly relies on rearview and side mirrors, which often have blind spots or blind areas and require manual monitoring by the driver. In recent years, some blind spot monitoring methods have emerged in the passenger vehicle sector, utilizing sensors such as radar, cameras, and lidar combined with deep learning methods. These methods can effectively monitor blind spots, but they usually require high computing resources, making them difficult to apply to commercial vehicle controller chips.

[0003] This invention aims to provide a high-efficiency visual blind spot monitoring method based on a low-computing-power chip. On the one hand, it meets the national standards for blind spot monitoring, and on the other hand, it provides a technical solution to improve the driving safety and comfort of commercial vehicle drivers. Summary of the Invention

[0004] To address the problems existing in the prior art, the purpose of this invention is to provide a blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform.

[0005] The technical solution adopted by the present invention to solve its technical problem is: a blind spot monitoring system based on a low computing power processing unit of a commercial vehicle platform, including a blind spot range calibration module, an image processing module, a perception algorithm module, an information processing module, and an early warning module;

[0006] The blind zone range calibration module is used to define the location of the blind zone and to calibrate the position of pixels inside the camera using the blind zone range.

[0007] The image processing module captures camera data and performs image preprocessing based on the calibrated position;

[0008] The perception algorithm module receives real-time frame images processed by the image processing module and completes the deployment preparation of the information processing unit for the customized model.

[0009] The information processing module builds a blind spot monitoring model, receives processed real-time frame images, performs inference operations, and obtains the category and confidence information of the inference detection results.

[0010] The early warning module receives the calculation results from the information processing module and issues corresponding early warnings based on the pre-set blind spot monitoring and early warning broadcasting strategy and the confidence level of the blind spot detection results.

[0011] Preferably, the blind spot location is defined as including three blind spot areas: the front, left, and right sides of the vehicle body, and the camera pixel location is defined as including the front, left, and right cameras of the vehicle body.

[0012] Preferably, the blind spot range calibration module uses cameras installed around the vehicle body to collect real-time data on pedestrians or vehicles within the blind spot range and transmits the collected blind spot target information to the image processing module.

[0013] Preferably, the image processing module includes receiving the acquired image data frame information and segmenting the image according to the ROI region; image preprocessing is to sparsify and normalize the captured data frame image data.

[0014] Preferably, the perception algorithm module identifies the types and locations of pedestrians and vehicles using a convolutional neural network.

[0015] Preferably, the perception algorithm module uses New CSP-DarkNet-53 to analyze the acquired image data frames to obtain target information, and by analyzing the operators with high processing efficiency of the commercial vehicle platform's computing power unit, it equivalently replaces some operators in the convolutional neural network.

[0016] Preferably, the perception algorithm module includes model building, model quantization, network pruning, and model sparsification techniques based on multi-scale feature fusion of low-computing-power processing units, which reduces the complexity of model inference and enables stable operation on low-computing-power resource platforms.

[0017] Preferably, the information processing module includes a low-computing-power processing unit for model deployment and inference, which uses parallel computing to construct a detection model and transmits the real-time inference results to the early warning module.

[0018] Preferably, the warning module includes a set of in-vehicle voice broadcasting units for receiving the calculation results from the information processing module.

[0019] The present invention has the following beneficial effects:

[0020] 1) By calibrating the pixel position of the camera blind zone, the ROI blind zone is actively defined, reducing the input size of the convolutional neural network in the perception algorithm module, reducing the computational load of the low-computing-power processing unit in the information processing module, and constructing a lightweight network detection model based on the low-computing-power processing unit for the selected pixel area.

[0021] 2) Combining model quantization, model sparsification and model pruning techniques to further compress the model and improve the real-time detection frame rate. Furthermore, to address the problem of low efficiency in the serial execution of image preprocessing, detection inference, and detection result judgment and transmission operations, a multi-threaded parallel execution system is designed using the concept of parallel computing to improve the comprehensive detection efficiency of multi-frame detection and the real-time monitoring frame rate of multiple cameras.

[0022] 3) In the warning prompt module, a customized prompt sound is broadcast by combining the preset blind spot monitoring warning broadcast strategy with the confidence level of the blind spot detection results. This provides drivers with warning prompts at different warning levels. At the same time, a corresponding display screen flashing prompt interface is designed to provide comprehensive prompts for the driver's vision and hearing, effectively reducing the potential risk of drivers ignoring blind spot warning information and effectively improving the driving safety of the vehicle. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall structure of the present invention.

[0024] Figure 2 This is a schematic diagram of the CNN blind spot monitoring model structure provided in the embodiment.

[0025] Figure 3 This is a schematic diagram of the blind spot provided in the embodiment. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, a blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform includes a blind spot range calibration module, an image processing module, a perception algorithm module, an information processing module, and a warning prompt module.

[0028] The blind spot calibration module includes the delineation of blind spot locations and the calibration of the internal pixel positions of the cameras using the blind spot range. The delineation of blind spot locations includes three blind spot ranges: the front, left, and right sides of the vehicle body. The camera pixel position calibration includes three sets of monocular fisheye cameras for the front, left, and right sides of the vehicle body. The information acquisition of the blind spot calibration module includes cameras installed around the vehicle body to collect real-time data on pedestrians, vehicles, and other targets within the blind spot range, and transmits the collected blind spot target information to the image processing module.

[0029] The image processing module includes a set of computing units for capturing camera data and performing image preprocessing operations based on the calibrated position, including segmentation, sparsification, and image normalization operations based on the ROI region. The image processing module includes receiving the acquired image data frame information and segmenting the image according to the ROI region. The image preprocessing operations include sparsification and normalization operations on the captured data frame image data to reduce computational complexity.

[0030] The perception algorithm module includes customized blind spot monitoring model design based on low-computing-power processing units, such as detection model construction, model network sparsification, model pruning, and model quantization. It is used to receive real-time frame images processed by the image processing module and complete the deployment preparation of the customized model's information processing unit. The perception algorithm module includes the identification of pedestrian and vehicle types and locations through convolutional neural networks. The perception algorithm module includes the use of New CSP-DarkNet-53 to analyze data frames collected by the acquisition unit to obtain target information. By analyzing operators with high processing efficiency of the commercial vehicle platform's computing power unit, equivalent replacements are performed on some operators in the convolutional neural network. The perception algorithm module includes model construction, model quantization, network pruning, and model sparsification techniques based on low-computing-power processing units. By reducing the model inference complexity, it achieves stable operation on low-computing-power resource platforms.

[0031] The information processing module includes a set of information computing and processing units for building a customized blind spot monitoring model system engineering. It receives the processed real-time frame images, performs inference operations, and obtains the category and confidence information of the inference detection results. The information processing module includes a platform low-computing power processing unit for model deployment and inference. It uses parallel computing to build the detection model inference engineering and transmits the real-time inference results to the early warning module.

[0032] The warning and alert module includes a set of in-vehicle voice broadcasting units, which receive the calculation results of the information processing module. Through a preset blind spot monitoring warning and alert broadcasting strategy combined with the confidence level of the blind spot detection results, it implements customized alert voice broadcasts, providing drivers with warning alerts of different levels, reducing the potential risk of drivers ignoring blind spot warning information.

[0033] Example

[0034] like Figure 2-3 As shown, the blind zone calibration module mainly includes calibrating the blind zone monitoring camera to achieve distortion correction. Based on a checkerboard calibration board, multiple shots are taken at different distances using the calibration board while keeping the camera's field of view constant. OpenCV functions are used for calibration to obtain the camera's intrinsic parameters and distortion coefficients, and the obtained parameters are then used for distortion correction.

[0035] The image processing module is used to capture camera data and perform image preprocessing operations based on the calibrated position. Based on the complexity of pixel edges within the monitoring range, the region of interest (ROI) is initially obtained. When there is no target on the road surface, there are only a few pixel edges within the monitoring range; conversely, there are denser edge lines. Sparsity reduction and image normalization operations are then performed based on the ROI.

[0036] The perception algorithm module receives blind spot perception information from the camera and determines the blind spot perception result using image processing and deep learning algorithms. A CNN blind spot detection model is used to detect target objects in the region of interest (ROI), where the target objects include pedestrians, motor vehicles, and non-motor vehicles. Figure 2 As shown, the CNN blind spot detection model structure includes a backbone network for feature extraction, a Neck multi-scale feature fusion network, and a Head target detection head, connected in sequence. The feature extraction backbone network uses New CSP-Darknet53. The Neck layer uses FPN and PAN modules. After top-down feature fusion by the FPN module, bottom-up feature fusion by the PAN module is performed to enhance the localization capability at multiple scales. The Head uses the K-means method to cluster the detection boxes in the training set data to obtain Anchors, which are used to generate detection boxes during prediction. A fully connected layer with a Softmax function is used to classify the features of each detection box. Finally, Regression is used to regress each detection box to obtain its position and size.

[0037] (1) Construct a blind spot monitoring dataset by collecting video data from cameras within the vehicle's blind spot, including pedestrians, motor vehicles, and non-motorized vehicles, using the acquisition devices on the target vehicle. Frames are extracted from the collected video data, and the target objects in each extracted image are labeled to obtain the blind spot monitoring dataset. The arrangement of the acquisition devices is as follows: Figure 3 As shown, 1 is the front-view camera, 2 and 3 are the left front and left rear cameras respectively, and 4 and 5 are the right front and right rear cameras respectively.

[0038] (2) End-to-end model training optimization: Based on the blind spot monitoring dataset, the training set, test set, and validation set are divided in an 8:1:1 ratio. Load the training dataset and initialize the hyperparameters, which include the learning rate, optimizer, number of iterations, IOU threshold, and anchor threshold.

[0039] (3) Iterative optimization of the model: the Mosaic method is used for image augmentation. The augmented image data is used as input to calculate the model prediction result and the label error value. The error value is the target detection loss.

[0040] The object detection loss is a weighted sum of the classification loss, objectness loss, and location loss. The calculation formula is:

[0041] Loss=λ1L cls +λ2L obj +λ3L loc

[0042] Lambda is the balance coefficient, which is 0.5, 1 and 0.05 respectively.

[0043] The classification loss is calculated using the binary cross-entropy function. The formula is:

[0044]

[0045] Where y is the label corresponding to the input sample (1 for positive samples and 0 for negative samples), and p is the probability that the model predicts the input sample to be a positive sample.

[0046] Localization loss measures the degree of overlap between predicted and ground truth bounding boxes in object detection. Assuming the predicted bounding box is A and the ground truth bounding box is B, its calculation formula is:

[0047]

[0048] ADAM is used to optimize the error of the loss function during model training.

[0049] (4) Model accuracy evaluation: Load model weights, read test dataset, evaluate model recognition accuracy, target detection task indicators include average accuracy at an intersection-union ratio of 0.5.

[0050] After completing the construction of the CNN blind spot monitoring model, the model algorithm is deployed and inferred. The CNN blind spot monitoring model is pruned and compressed, and int8 model quantization is performed.

[0051] The information processing module primarily provides computational support for the blind spot perception model, improving its inference speed and accuracy. The CNN blind spot detection model is deployed to an edge controller for multi-threaded inference. Multi-threading is used to accelerate the detection of target objects located within the blind spot of the target vehicle from multiple cameras.

[0052] The warning and alert module includes a set of in-vehicle voice broadcast units used to receive the calculation results from the information processing module. For example... Figure 3As shown, the lateral detection range is 0m-3m from the left and right sides of the vehicle (outer edge of the wheels); the longitudinal detection range is 3m in front of the rearview mirror and 22m behind the rearview mirror. When the vehicle is not using the turn signal, if a target object with a relative speed of 1-30km / h is detected in the alarm area, a level one alarm is triggered, and a level one warning alarm signal is sent to the CAN bus. The voice broadcast unit continuously broadcasts "Please note, an object has entered the blind spot," and the warning interface flashes on the vehicle display screen. When the vehicle is using the corresponding turn signal, if a target object with a relative speed of 1-30km / h is detected in the corresponding alarm area, a level two alarm is triggered, and a level two warning alarm signal is sent to the CAN bus. The voice broadcast unit continuously broadcasts "Please note, an object has entered the turning blind spot," and the warning interface flashes on the vehicle display screen.

[0053] This invention is not limited to the above-described embodiments. Anyone should know that any structural changes made under the guidance of this invention, and any technical solutions that are the same as or similar to this invention, fall within the protection scope of this invention.

[0054] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform, characterized in that, It includes a blind zone calibration module, an image processing module, a perception algorithm module, an information processing module, and an early warning module; The blind zone range calibration module is used to define the location of the blind zone and to calibrate the position of pixels inside the camera using the blind zone range. The image processing module captures camera data and performs image preprocessing based on the calibrated position; The perception algorithm module receives real-time frame images processed by the image processing module and completes the deployment preparation of the information processing unit for the customized model. The information processing module builds a blind spot monitoring model, receives processed real-time frame images, performs inference operations, and obtains the category and confidence information of the inference detection results. The early warning module receives the calculation results from the information processing module and issues corresponding early warnings based on the pre-set blind spot monitoring and early warning broadcasting strategy and the confidence level of the blind spot detection results.

2. The blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform according to claim 1, characterized in that, The blind spot location is defined as three blind spot areas: the front, left, and right sides of the vehicle body. The camera pixel location is defined as the front, left, and right cameras of the vehicle body.

3. The blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform according to claim 1, characterized in that, The blind spot range calibration module uses cameras installed around the vehicle body to collect real-time data on pedestrians or vehicles within the blind spot range and transmits the collected blind spot target information to the image processing module.

4. The blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform according to claim 1, characterized in that, The image processing module includes receiving the acquired image data frame information and segmenting the image according to the ROI region; image preprocessing is to sparsify and normalize the captured data frame image data.

5. The blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform according to claim 1, characterized in that, The perception algorithm module uses a convolutional neural network to identify the types and locations of pedestrians and vehicles.

6. The blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform according to claim 5, characterized in that, The perception algorithm module uses New CSP-DarkNet-53 to analyze the acquired image data frames to obtain target information. By analyzing the operators with high processing efficiency of the commercial vehicle platform's computing power unit, it equivalently replaces some operators in the convolutional neural network.

7. The blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform according to claim 5, characterized in that, The perception algorithm module includes model building, model quantization, network pruning, and model sparsification techniques based on multi-scale feature fusion of low-computing-power processing units. By reducing the complexity of model inference, it achieves stable operation on low-computing-power resource platforms.

8. The blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform according to claim 1, characterized in that, The information processing module includes a low-computing-power processing unit for model deployment and inference, which uses parallel computing to build a detection model and transmits the real-time inference results to the early warning module.

9. The blind spot monitoring system based on a low-computing-power processing unit of a commercial vehicle platform according to claim 1, characterized in that, The warning module includes a set of in-vehicle voice broadcasting units for receiving the calculation results from the information processing module.

Images