A blind spot warning method and device based on a sanitation vehicle surround vision system

Abstract

The present invention discloses a blind spot warning method and device based on the surround view system of a sanitation vehicle. A visual odometer is constructed through the surround view camera of the sanitation vehicle, and scale recovery calculation is completed using multi-camera information to achieve three-dimensional perception of the blind spot of the sanitation vehicle. Subsequently, a spatial hash table and a Euclidean clustering algorithm are used to achieve rapid processing of three-dimensional point cloud data, and a collision detection algorithm based on the separating axis theorem is used to achieve a vehicle blind spot detection function, thereby balancing the contradiction between calculation speed and calculation accuracy, and ensuring real-time warning of blind spots around the vehicle by sanitation vehicles under various complex working conditions. The method has high practical value.

Description

Technical Field

[0001] The invention relates to a blind spot warning method and device, in particular to a blind spot warning method and device based on a surround vision system of a sanitation vehicle, belonging to the field of vehicle anti-collision warning. Background Art

[0002] Sanitation vehicles usually face many tedious and time-consuming tasks, such as road cleaning, watering, dust removal, and transportation. Currently, sanitation vehicles mainly rely on manual drivers to observe and complete the corresponding tasks. This method makes it easy for drivers to have inattention and fatigue driving, which affects driving safety. Advanced driver assistance systems and autonomous driving are playing an increasingly important role in ensuring road traffic safety. Vehicle blind spot monitoring is an important part of environmental perception research. It effectively reduces the safety hazards caused by visual blind spots or inattention of drivers. It is an important part of vehicle assisted driving technology, which further ensures the safety of vehicle driving.

[0003] At present, there are mainly two types of vehicle blind spot detection and monitoring solutions:

[0004] The first solution is such as Chinese invention patent 202111572237.3. This solution provides a vehicle collision avoidance perception algorithm based on computer vision. The system includes: during the driving process of the vehicle, pictures taken by multiple cameras around the vehicle body are obtained at the same time, and the deformity of multiple pictures is corrected, and multiple pictures are spliced ​​based on multiple image stitching technology, and the image information after stitching is extracted, and then the target in the picture is identified and tracked, and finally the path of the target in the picture is predicted and risk inference is performed. However, this solution has the following shortcomings: First, this solution uses multiple image stitching technology to obtain information about the surrounding area of ​​the vehicle, and image stitching technology has high requirements on the calibration accuracy of the camera. At the same time, a large amount of computing resources are required during the image stitching process, which increases the hardware requirements of the system. Second, this solution uses object recognition methods for tracking, which will cause missed detection problems for unknown objects, causing safety hazards.

[0005] The second solution is as shown in Chinese invention patent 201910845641.X. This solution provides a traffic collision avoidance method based on computer vision human body recognition. First, the computer vision recognition terminal installed on the car extracts and recognizes the human head in the video in real time according to the captured video. Since the pixel ratio of the human head in the video image is inversely correlated with its distance from the computer vision recognition terminal, the distance between it and the car can be determined according to the pixel ratio. When the distance between the human body and the car is within a certain distance, the recognition signal is fed back to the driver and the car collision avoidance and impact reduction program is started to realize the visual collision avoidance function. However, this solution has the following shortcomings: First, this solution uses the object recognition method to track the human body, which will cause missed detection problems for unknown objects, causing safety hazards. At the same time, the object recognition of multiple cameras requires a lot of computing resources, which increases the hardware requirements of the system. Second, this solution uses the pixel ratio occupied by the target object to estimate the distance information. Since the camera angle, human body angle and human head size will affect the distance calculation result, the distance estimated by this method has low accuracy. When the vehicle is driving fast, it is easy to cause misjudgment and affect the safety of the vehicle. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a blind spot warning method and device based on a surround vision system of a sanitation vehicle, so as to improve the accuracy and robustness of blind spot obstacle detection.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A blind spot warning method based on a sanitation vehicle surround vision system, characterized by comprising the following steps:

[0009] Step 1: Collect the blind spot video stream of the sanitation vehicle through the vehicle's multi-channel blind spot cameras;

[0010] Step 2: Process the collected video stream and construct a local map through visual odometry calculation method;

[0011] Step 3: Restore the scale of the visual odometer based on the calibration parameters of the surround camera;

[0012] Step 4: Use the spatial hash table to reconstruct and cluster the point cloud data to generate a two-dimensional rectangular clustering frame;

[0013] Step 5: Generate a vehicle rectangular frame based on the size information of the sanitation vehicle, and use the collision detection algorithm to realize blind spot detection;

[0014] Step 6: Implement blind spot warning for sanitation vehicles based on vehicle speed and blind spot detection results and inform the sanitation vehicle driver via voice or video.

[0015] Furthermore, the step 2 is specifically as follows:

[0016] 2.1. Extract key points from each surround-view camera image, use the calibration parameters of the surround-view camera as initial values ​​and add them to the odometer optimization system, build a visual odometer calculation method in the vehicle coordinate system, and continuously track the vehicle blind spot;

[0017] 2.2. Maintain 5 to 9 frames of local maps to meet blind spot detection requirements.

[0018] Furthermore, the step 2.1 is specifically as follows:

[0019] After the odometer is initialized, according to the external parameters of each blind spot camera , the map point in the vehicle coordinate system Projected into the corresponding image, thus building a unified optimization framework, the overall residual function is as follows:

[0020]

[0021] Among them, the first part on the right side of the equation is the prior error, the second part is the reprojection error, the function ρ is the robust kernel function, and the function is the prior error function, for The Jacobian matrix for the optimization variable X, is the covariance matrix of the prior error, function is the visual odometry reprojection error function, is the camera observation value, X is the variable to be optimized, and there are: ,in , , are the translation and rotation from the car system to the world system at time i, is the pose matrix of the camera relative to the vehicle body, that is, the external parameter of the camera, is the inverse depth value of the corner point, is the covariance matrix of the reprojection error;

[0022] Among them, the visual odometry reprojection error function is:

[0023] ,

[0024] in, For map points The normalized coordinates in the car system are: For map points The coordinates in the vehicle system obtained by the camera projection model;

[0025] And there are:

[0026] ,

[0027] in, For map points Normalized coordinates in the car system, d map point The inverse depth value in the car system, and are the pose matrices of the camera at time i and j in the world system, , , and is the parameter to be optimized;

[0028] Thus, the video stream data of multiple blind spot cameras are unified into an optimization framework for calculation, and the first nonlinear optimization algorithm is used for iterative solution, finally realizing the operation of the visual odometer under the surround view camera.

[0029] Furthermore, the first nonlinear optimization algorithm is a Levenberg-Marquardt algorithm.

[0030] Furthermore, the step three is specifically as follows:

[0031] 3.1. Select a pair of cameras with the most overlapping fields of view from the surround cameras, and define them as the left camera and the right camera respectively;

[0032] 3.2. Project the key points extracted from the left camera to the right camera, and construct an optimization algorithm for scale calculation based on the existing calibration parameters of the two cameras;

[0033] 3.3. Use the calculated scale value to restore the scale of camera pose and point cloud.

[0034] Furthermore, the step 3.2 is specifically as follows:

[0035] According to the external parameters between the blind spot cameras , the key points extracted from the left camera Project it to the right camera to construct the photometric error function, which is as follows:

[0036]

[0037] Among them, function ρ is the robust kernel function, function and Key points can be obtained The pixel value at the location, function and is the camera projection model function, For key points In the camera system, s is the variable to be optimized, i.e., the scale value, and γ is the covariance matrix of the photometric error;

[0038] According to the function Get the key points Coordinates in the left camera system , by the function Get at the key point The pixel coordinates under the right camera are: ,in:

[0039] , K is the intrinsic parameter matrix of the camera, , and Represents the rotation and translation matrices between the left and right cameras respectively;

[0040] Therefore, the scale calculation is completed by adopting a loosely coupled strategy, and the second nonlinear optimization algorithm is used for iterative solution to finally achieve scale recovery.

[0041] Furthermore, the second nonlinear optimization algorithm adopts a Gauss-Newton algorithm.

[0042] Furthermore, the step 4 is specifically as follows:

[0043] 4.1. Use the RANSAC-based plane fitting algorithm to preprocess the point cloud data in the local map to remove the ground point cloud data;

[0044] 4.2. Calculate the hash index according to the spatial position of each point cloud and insert it into the spatial hash data structure;

[0045] 4.3. Perform clustering calculation on the point cloud in the hash data structure to generate a two-dimensional rectangular clustering box in the vehicle coordinate system.

[0046] Furthermore, the step five is specifically as follows:

[0047] 5.1. Generate a vehicle rectangular frame based on the size information of the sanitation vehicle. At the same time, in order to ensure the safety of the vehicle, increase the expansion coefficient on the basis of the vehicle rectangular frame to enlarge it;

[0048] 5.2. Based on the vehicle rectangular frame and point cloud clustering frame, use the collision detection algorithm to detect whether the vehicle will collide.

[0049] Furthermore, the step six is ​​specifically as follows:

[0050] 6.1. Make a comprehensive judgment based on the vehicle's driving status and blind spot detection results to analyze whether there is a collision risk;

[0051] 6.2. If there is a risk of collision, an alarm will be issued via voice or video to notify the sanitation vehicle driver for emergency treatment.

[0052] A blind spot warning device based on a surround view system of a sanitation vehicle, characterized in that it comprises a multi-channel blind spot camera, an on-board intelligent computing box, a vehicle speed sensor, an alarm unit and a display screen, the on-board intelligent computing box having a built-in blind spot warning method program based on the surround view system of a sanitation vehicle, the multi-channel blind spot camera is connected to the on-board intelligent computing box for obtaining a vehicle surround view video stream, the vehicle speed sensor is connected to the on-board intelligent computing box for obtaining a vehicle driving status, the alarm unit is connected to the on-board intelligent computing box for issuing a collision alarm, and the display screen is connected to the on-board intelligent computing box for displaying a blind spot collision detection result of the on-board intelligent computing box.

[0053] Compared with the prior art, the present invention has the following advantages and effects:

[0054] 1. The present invention adopts direct visual mileage calculation method to monitor the blind spots of sanitation vehicles. Compared with the object recognition method, it effectively reduces the false detection and missed detection caused by camera field of view, unknown objects, etc., and significantly improves the accuracy and robustness of blind spot obstacle depth perception;

[0055] 2. The present invention effectively solves the problem of large visual blind spots of sanitation vehicles by constructing a surround view system through multiple blind spot cameras. At the same time, a visual mileage calculation method is constructed in the vehicle coordinate system, and the camera external parameters are used as one of the optimization parameters to participate in the global optimization, thereby improving the problem of difficulty in splicing different camera images in the surround view system;

[0056] 3. The present invention utilizes multi-camera information and adopts a loosely coupled strategy to complete scale recovery, obtains the real scale information of the vehicle driving scene, ensures the accuracy of blind spot obstacle distance detection, maintains the robustness of the monocular visual mileage calculation method, and reduces the loss during tracking;

[0057] 4. The present invention uses a spatial hash table data structure and a clustering algorithm to achieve rapid processing of point cloud data, reduces the computational complexity of the collision detection algorithm, and ensures the real-time performance of blind spot detection by multiple surround view cameras at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of a blind spot warning method based on a surround vision system of a sanitation vehicle of the present invention.

[0059] Figure 2 4 is a flow chart of visual odometer scale recovery according to an embodiment of the present invention.

[0060] Figure 3 Schematic diagram of blind spot detection according to an embodiment of the present invention.

[0061] Figure 4 It is a module schematic diagram of a blind spot warning device based on a surround vision system of a sanitation vehicle of the present invention. DETAILED DESCRIPTION

[0062] In order to elaborate on the technical scheme adopted by the present invention to achieve the predetermined technical purpose, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments, and the technical means or technical features in the embodiments of the present invention can be replaced without paying creative work. The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0063] like Figure 1 As shown, a blind spot warning method based on a surround view system of a sanitation vehicle of the present invention comprises the following steps:

[0064] Step 1: Collect the blind spot video stream of the sanitation vehicle through the vehicle's multi-channel blind spot cameras.

[0065] For ease of description, assume that the camera coordinate system conforms to the right-hand coordinate system, the Z-axis direction is the optical axis of the camera, and the origin of the coordinate system is the optical center of the camera. Assume that the vehicle coordinate system conforms to the right-hand coordinate system, the X-axis direction is the center axis of the vehicle and points to the front of the vehicle, the Z-axis is perpendicular to the XY plane of the frame and points to the sky, and the origin of the coordinate system is located between the XY plane of the frame and the center of the vehicle.

[0066] Step 2: Process the collected video stream and construct a local map through visual odometry.

[0067] The visual odometry method is used to reconstruct the vehicle's blind spot in three dimensions. Compared with the visual SLAM technology, the visual odometry method does not include its global optimization and mapping, which significantly reduces the algorithm's computational complexity and system memory requirements. According to the algorithm principle, the traditional visual odometry can be divided into feature point method, semi-direct method, direct point and multi-sensor fusion method. The corresponding typical open source algorithms include the front end of ORB-SLAM2, SVO, DSO and VIO.

[0068] Step 2 is as follows:

[0069] 2.1. Extract key points from each surround-view camera image, use the calibration parameters of the surround-view camera as initial values ​​and add them to the odometer optimization system, build a visual odometer calculation method in the vehicle coordinate system, and continuously track the vehicle blind spot.

[0070] Traditional visual odometry can usually only process data from a monocular camera or a binocular camera, and with the addition of a binocular camera, the amount of calculation often increases significantly. Therefore, improvements are made to the traditional visual odometry so that it can process data from multiple cameras at the same time and keep the amount of calculation within a reasonable range.

[0071] Therefore, in the present invention, after the odometer is initialized, the external parameters of each blind spot camera are , the map point in the vehicle coordinate system Projected into the corresponding image, thus building a unified optimization framework, the overall residual function is as follows:

[0072]

[0073] Among them, the first part on the right side of the equation is the prior error, the second part is the reprojection error, the function ρ is the robust kernel function, and the function is the prior error function, for The Jacobian matrix for the optimization variable X, is the covariance matrix of the prior error, function is the visual odometry reprojection error function, is the camera observation value, X is the variable to be optimized, and there are: ,in , , are the translation and rotation from the car system to the world system at time i, is the pose matrix of the camera relative to the vehicle body, that is, the external parameter of the camera, is the inverse depth value of the corner point, is the covariance matrix of the reprojection error;

[0074] Among them, the visual odometry reprojection error function is:

[0075] ,

[0076] in, For map points The normalized coordinates in the car system are: For map points The coordinates in the vehicle system obtained by the camera projection model;

[0077] And there are:

[0078] ,

[0079] in, For map points Normalized coordinates in the car system, d map point The inverse depth value in the car system, and are the pose matrices of the camera at time i and j in the world system, , , and is the parameter to be optimized;

[0080] Thus, the video stream data of multiple blind spot cameras are unified into an optimization framework for calculation, and the first nonlinear optimization algorithm is used for iterative solution, wherein the first nonlinear optimization algorithm preferably uses the Levenberg-Marquardt algorithm. Finally, the operation of the visual odometer under the surround camera is realized, while the repeated extraction of corner points is avoided, the amount of calculation is reduced, and the real-time performance of the algorithm is ensured.

[0081] In the present invention, key points are extracted from each surround camera image, and the key points are projected using camera extrinsics to construct a visual odometer in the vehicle coordinate system to continuously track the vehicle blind spot.

[0082] The visual odometry uses the DSO framework. DSO is a sparse direct method of visual odometry. In contrast, SVO is a semi-direct method, and ORB-SLAM2 is a pure feature method. DSO puts data association and pose estimation into a unified nonlinear optimization problem. Its objective function is to minimize the photometric error. While maintaining the same or higher accuracy, it can reach five times the speed of the traditional feature point method. From the back-end point of view, DSO uses a sliding window consisting of 5 to 7 key frames as its back-end. This window exists throughout the VO process. By maintaining and updating the Hessian matrix and b vector in the Levenberg-Marquardt method, the pose and map points are optimized through marginalization operations. At the same time, in order to make the algorithm more robust, DSO proposes photometric calibration and dynamically estimates photometric parameters during optimization, which will have better performance for image brightness changes caused by different camera exposures.

[0083] Since the DSO visual odometry is mainly aimed at monocular cameras, the external parameters of the surround camera are used to project the key points extracted by the camera between cameras, fused into a whole key point data and added to the optimization system. Since the camera external parameter error will affect the accuracy of key point projection and pose estimation, the camera external parameters are also added to the optimization system, thereby improving the problem of difficulty in stitching different camera images in the surround system.

[0084] 2.2. Since the visual mileage calculation method does not maintain a global map, and blind spot warning requires the extraction of a certain number of three-dimensional point clouds, 5 to 9 frames of local maps are maintained in the optimization algorithm of 2.1 to meet the blind spot detection requirements.

[0085] Step 3: Restore the scale of the visual odometer based on the calibration parameters of the surround camera. Figure 2 As shown in Figure 1, a pair of cameras with the largest overlap in field of view is selected from the surround view cameras for scale optimization. The specific process is as follows:

[0086] 3.1. Select a pair of cameras with the most overlapping fields of view from the surround-view cameras and define them as the left camera and the right camera respectively.

[0087] 3.2. Project the key points extracted from the left camera to the right camera, and construct a scale optimization algorithm based on the existing calibration parameters of the two cameras to perform scale calculation.

[0088] Traditional scale calculation usually adopts a tightly coupled solution, such as stereo matching. Although the calculation accuracy of this solution is high, it will reduce the robustness of the monocular visual odometer and significantly increase the calculation amount of the odometer. Therefore, based on the traditional scale calculation, improvements are made and a loosely coupled calculation strategy is adopted. This solution has higher calculation efficiency and significantly reduces the calculation burden of stereo vision. At the same time, the calculation accuracy of this solution is not significantly reduced, ensuring the accuracy and real-time performance of the algorithm.

[0089] In the present invention, according to the external parameters between the blind spot cameras , the key points extracted from the left camera Project it to the right camera to construct the photometric error function, which is as follows:

[0090]

[0091] Among them, function ρ is the robust kernel function, function and Key points can be obtained The pixel value at the location, function and is the camera projection model function, For key points In the camera system, s is the variable to be optimized, i.e., the scale value, and γ is the covariance matrix of the photometric error;

[0092] According to the function Get the key points Coordinates in the left camera system , by the function Get at the key point The pixel coordinates under the right camera are: ,in:

[0093] , K is the intrinsic parameter matrix of the camera, , and Represents the rotation and translation matrices between the left and right cameras respectively;

[0094] Thus, the scale calculation is completed by adopting a loose coupling strategy, and the second nonlinear optimization algorithm is adopted for iterative solution, wherein the second nonlinear optimization algorithm is preferably the Gauss-Newton algorithm. Finally, the scale recovery is realized, the amount of calculation is reduced, and the real-time performance of the algorithm is ensured.

[0095] 3.3. Use the calculated scale value to restore the scale of camera pose and point cloud.

[0096] The visual odometry operation and scale optimization operation adopt a loose coupling strategy, and do not affect each other in their respective calculation processes. The visual odometry provides point cloud data for scale optimization, and the scale value obtained by the scale optimization calculation is returned to the visual odometry for camera pose and point cloud scale recovery. Therefore, the failure of scale optimization calculation will not affect the normal operation of the visual odometry.

[0097] Step 4: Use the spatial hash table to reconstruct and cluster the point cloud data to generate a two-dimensional rectangular clustering box.

[0098] 4.1. Use the RANSAC-based plane fitting algorithm to perform plane fitting on the ground point cloud. According to the calibrated camera extrinsics, the normal vector of the horizontal road surface in the vehicle coordinate system can be calculated. Therefore, in order to improve the stability of the fitting algorithm, the prior road surface normal vector parameters are used to constrain the fitting algorithm, thereby removing the ground point cloud data.

[0099] 4.2. Calculate the hash index according to the spatial position of each point cloud and insert it into the spatial hash data structure.

[0100] 4.3. Perform clustering calculation on the point cloud in the hash data structure to generate a two-dimensional rectangular clustering frame in the vehicle coordinate system. The clustering algorithm uses Euclidean clustering, which groups the point cloud into point cloud clusters. The projection distance between any two points in the point cloud cluster is less than a certain threshold. The projection distance refers to the distance between two points projected onto the XY plane of the vehicle frame.

[0101] Step 5: Generate a vehicle rectangular frame based on the size information of the sanitation vehicle and use the collision detection algorithm to achieve blind spot detection.

[0102] 5.1. Generate a vehicle rectangular frame based on the sanitation vehicle size information (including vehicle length and width size information). At the same time, in order to ensure vehicle safety, increase the expansion coefficient on the basis of the vehicle rectangular frame to appropriately enlarge it, and the enlargement ratio is about 1.1 times.

[0103] 5.2. Based on the vehicle rectangular frame and point cloud clustering frame, use the collision detection algorithm to detect whether the vehicle will collide.

[0104] like Figure 3 As shown in the figure, the rectangular box in the middle of the grid is the vehicle, and the numbers in the box are the vehicle coordinates. The other boxes are the obstacle rectangular boxes obtained by clustering, and the numbers in the boxes are the distances from the obstacle rectangular boxes to the center of the vehicle. In this scenario, all the boxes obtained are rectangular boxes, so the collision detection algorithm uses the Separating Axis Theorem (SAT) algorithm, which is an algorithm for detecting collisions between convex polygons. It determines whether two convex polygons collide by detecting and judging whether there is a gap between them. The algorithm has the characteristics of fast operation speed and accurate judgment, which ensures the real-time performance of collision detection.

[0105] Step 6: Implement blind spot warning for sanitation vehicles based on vehicle speed and blind spot detection results and inform the sanitation vehicle driver via voice or video.

[0106] 6.1. Make a comprehensive judgment based on the vehicle's driving status and blind spot detection results to analyze whether there is a collision risk.

[0107] 6.2. If there is a collision risk, an alarm will be issued via voice or video according to the urgency of the collision risk, notifying the sanitation vehicle driver to take emergency measures and achieving continuous monitoring of the blind spots of sanitation vehicles.

[0108] like Figure 4 As shown, a blind spot warning device based on the surround view system of a sanitation vehicle includes multiple blind spot cameras, an on-board intelligent computing box, a vehicle speed sensor, an alarm unit and a display screen. The on-board intelligent computing box has a built-in blind spot warning method program based on the surround view system of a sanitation vehicle. The multiple blind spot cameras are connected to the on-board intelligent computing box to obtain the vehicle surround view video stream, the vehicle speed sensor is connected to the on-board intelligent computing box to obtain the vehicle driving status, the alarm unit is connected to the on-board intelligent computing box to issue a collision alarm, and the display screen is connected to the on-board intelligent computing box to display the blind spot collision detection result of the on-board intelligent computing box.

[0109] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A blind spot warning method based on a sanitation vehicle surround vision system, characterized in that The following steps are involved: Step 1: Collect the blind spot video stream of the sanitation vehicle through the vehicle's multi-channel blind spot cameras; Step 2: Process the collected video stream and construct a local map through visual odometry calculation method; Step 3: Restore the scale of the visual odometer based on the calibration parameters of the surround camera; The step three is specifically as follows: 3.

1. Select a pair of cameras with the most overlapping fields of view from the surround cameras, and define them as the left camera and the right camera respectively; 3.

2. Project the key points extracted from the left camera to the right camera, and construct an optimization algorithm for scale calculation based on the existing calibration parameters of the two cameras; The step 3.2 is specifically as follows: According to the external parameters T between the blind spot cameras stereo , the key point p extracted from the left camera i Project it to the right camera to construct the photometric error function, which is as follows: Among them, function ρ is a robust kernel function, and functions I1 and I0 can obtain the key point p i The pixel value at the location, functions r0 and r1 are camera projection model functions, The key point p i In the camera system, s is the variable to be optimized, i.e., the scale value, and γ is the covariance matrix of the photometric error; According to the function r0, the key point p can be obtained i Coordinate P in the left camera system i =[x,y,z] T , The function r1 is used to obtain the key point p i The pixel coordinates under the right camera are: in: [x′, y′, z′] T =K·(s·R stereo [x, y, z] T +t stereo ), K is the intrinsic parameter matrix of the camera, R stereo With t stereo Represents the rotation and translation matrices between the left and right cameras respectively; Thus, the scale calculation is completed by adopting a loosely coupled strategy, and the second nonlinear optimization algorithm is used for iterative solution, and finally the scale recovery is achieved; 3.

3. Use the calculated scale value to restore the scale of the camera pose and point cloud; Step 4: Use the spatial hash table to reconstruct and cluster the point cloud data in the local map to generate a two-dimensional rectangular clustering frame; Step 5: Generate a vehicle rectangular frame based on the size information of the sanitation vehicle, and use the collision detection algorithm to realize blind spot detection; Step 6: Implement blind spot warning for sanitation vehicles based on vehicle speed and blind spot detection results and inform the sanitation vehicle driver via voice or video.

2. A blind spot warning method based on a sanitation vehicle surround vision system according to claim 1, characterized in that: The step 2 is specifically as follows: 2.

1. Extract key points from each surround-view camera image, use the calibration parameters of the surround-view camera as initial values ​​and add them to the odometer optimization system, build a visual odometer calculation method in the vehicle coordinate system, and continuously track the vehicle blind spot; 2.

2. Maintain 5 to 9 frames of local maps to meet blind spot detection requirements.

3. A blind spot warning method based on a sanitation vehicle surround vision system according to claim 2, characterized in that: The step 2.1 is specifically as follows: After the odometer is initialized, according to the external parameters of each blind spot camera The map point under the vehicle coordinate system Projected into the corresponding image, thus building a unified optimization framework, the overall residual function is as follows: Among them, the first part on the right side of the equation is the prior error, the second part is the reprojection error, the function ρ is the robust kernel function, and the function r p is the prior error function, J P For r p The Jacobian matrix of the optimization variable X, ∑p is the covariance matrix of the prior error, and the function r f is the visual odometry reprojection error function, is the camera observation value, X is the variable to be optimized, and: X=[x n , x n+1 , ..., x n+N , d m , d m+1 , ..., d m+M ] T ,in are the translation and rotation from the car system to the world system at time i, is the pose matrix of the camera relative to the vehicle body, i.e., the external parameter of the camera, d i is the inverse depth value of the corner point, is the covariance matrix of the reprojection error; Among them, the visual odometry reprojection error function is: in, For map points The normalized coordinates in the car system are: For map points The coordinates in the vehicle system obtained by the camera projection model; And there are: in, For map points Normalized coordinates in the car system, d map point The inverse depth value in the car system, and are the pose matrices of the camera at time i and j in the world system, With d i is the parameter to be optimized; Thus, the video stream data of multiple blind spot cameras are unified into an optimization framework for calculation, and the first nonlinear optimization algorithm is used for iterative solution, finally realizing the operation of the visual odometer under the surround view camera.

4. The blind spot warning method based on the surround vision system of a sanitation vehicle according to claim 3 is characterized by: The first nonlinear optimization algorithm is the Levenberg-Marquardt algorithm.

5. The blind spot warning method based on the surround vision system of a sanitation vehicle according to claim 1, characterized in that: The second nonlinear optimization algorithm adopts the Gauss-Newton algorithm.

6. The blind spot warning method based on the surround vision system of a sanitation vehicle according to claim 1, characterized in that: The step 4 is specifically as follows: 4.

1. Use the RANSAC-based plane fitting algorithm to preprocess the point cloud data in the local map to remove the ground point cloud data; 4.

2. Calculate the hash index according to the spatial position of each point cloud and insert it into the spatial hash data structure; 4.

3. Perform clustering calculation on the point cloud in the hash data structure to generate a two-dimensional rectangular clustering box in the vehicle coordinate system.

7. The blind spot warning method based on the surround vision system of a sanitation vehicle according to claim 1, characterized in that: The step five is specifically as follows: 5.

1. Generate a vehicle rectangular frame based on the size information of the sanitation vehicle. At the same time, in order to ensure the safety of the vehicle, increase the expansion coefficient on the basis of the vehicle rectangular frame to enlarge it; 5.

2. Based on the vehicle rectangular frame and point cloud clustering frame, use the collision detection algorithm to detect whether the vehicle will collide.

8. The blind spot warning method based on the surround vision system of a sanitation vehicle according to claim 1, characterized in that: The step six is ​​specifically as follows: 6.

1. Make a comprehensive judgment based on the vehicle's driving status and blind spot detection results to analyze whether there is a collision risk; 6.

2. If there is a risk of collision, an alarm will be issued via voice or video to notify the sanitation vehicle driver for emergency treatment.

9. A blind spot warning device based on a sanitation vehicle surround vision system, characterized in that: It comprises multiple blind spot cameras, an on-board intelligent computing box, a vehicle speed sensor, an alarm unit and a display screen. The on-board intelligent computing box has a built-in blind spot warning method program based on the surround view system of a sanitation vehicle as described in any one of claims 1 to 8. The multiple blind spot cameras are connected to the on-board intelligent computing box to obtain the vehicle surround view video stream, the vehicle speed sensor is connected to the on-board intelligent computing box to obtain the vehicle driving status, the alarm unit is connected to the on-board intelligent computing box to issue a collision alarm, and the display screen is connected to the on-board intelligent computing box to display the blind spot collision detection result of the on-board intelligent computing box.

Images