A method and terminal for measuring distance between pedestrians in front of a vehicle based on binocular cameras

Abstract

The present invention provides a method and terminal for measuring distance of pedestrians in front of a vehicle based on a binocular camera, comprising: calibrating and stereo matching a binocular camera deployed at the top of the outer side of a vehicle, and training a pedestrian recognition model based on a set of pedestrian videos pre-shot by the binocular camera; starting the binocular camera, using the pedestrian recognition model to recognize pedestrian targets in the image in front of the vehicle shot in real time by the binocular camera, extracting the ROI area of ​​the pedestrian target on the current image in front of the vehicle if a pedestrian target is recognized, and obtaining the ROI depth map after removing the obstruction; randomly extracting multiple key nodes of the pedestrian target, and obtaining the depth information of each key node in the ROI depth map, using the mean of the depth information of each key node as the pedestrian distance, and displaying the image in front of the vehicle where the pedestrian target is recognized and the pedestrian distance in real time on the vehicle display screen. The present invention can quickly and accurately measure the distance between the vehicle and the target pedestrian when there is an obstruction in front of the pedestrian.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle automatic driving, and in particular to a method and terminal for measuring the distance between pedestrians in front of a vehicle based on a binocular camera. Background Art

[0002] Self-driving cars are the mainstream of future development in the automotive field. In the self-driving system based on visual navigation, target detection and ranging are the key technologies that determine the performance of the self-driving system.

[0003] The existing method of autonomous driving assisted target detection based on binocular camera stereo vision mainly obtains the depth map of the image by synchronously collecting the left and right images of the binocular camera and outputting the location information of the target obstacle.

[0004] However, this method consumes a lot of computing time, and when there are other obstacles in front of the target obstacle, the output position information will have errors. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method and terminal for measuring the distance between a vehicle and a pedestrian based on a binocular camera, which can quickly and accurately measure the distance between the vehicle and the target pedestrian when there is an obstruction in front of the pedestrian, thereby overcoming the defects of insufficient detection speed and insufficient detection accuracy in the existing pedestrian detection and ranging based on binocular cameras.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for measuring the distance between pedestrians in front of a vehicle based on a binocular camera comprises the following steps:

[0008] S1. Calibrate and stereo match the binocular camera deployed on the top of the outer side of the vehicle, and train a pedestrian recognition model based on a set of pedestrian videos pre-taken by the binocular camera;

[0009] S2, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera, if the pedestrian target is recognized, extracting the ROI area of ​​the pedestrian target on the current image in front of the vehicle, and obtaining the ROI depth map after removing the obstruction;

[0010] S3. Randomly extract multiple key nodes of the pedestrian target, and obtain the depth information of each key node in the ROI depth map, use the average of the depth information of each key node as the pedestrian distance, and display the vehicle front image in which the pedestrian target is identified and the pedestrian distance in real time on the vehicle display screen.

[0011] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0012] A vehicle-front pedestrian ranging terminal based on a binocular camera comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0013] S1. Calibrate and stereo match the binocular camera deployed on the top of the outer side of the vehicle, and train a pedestrian recognition model based on a set of pedestrian videos pre-taken by the binocular camera;

[0014] S2, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera, if the pedestrian target is recognized, extracting the ROI area of ​​the pedestrian target on the current image in front of the vehicle, and obtaining the ROI depth map after removing the obstruction;

[0015] S3. Randomly extract multiple key nodes of the pedestrian target, and obtain the depth information of each key node in the ROI depth map, use the average of the depth information of each key node as the pedestrian distance, and display the vehicle front image in which the pedestrian target is identified and the pedestrian distance in real time on the vehicle display screen.

[0016] The beneficial effects of the present invention are as follows: the present invention provides a method and terminal for measuring the distance between pedestrians in front of a vehicle based on a binocular camera. A binocular camera is deployed on the top of the outer side of a vehicle, and the binocular camera is calibrated, stereo matched, and a pedestrian recognition model is trained. When the vehicle is in the process of automatic driving, the pedestrian target can be recognized in time by the pedestrian recognition model on the image in front of the vehicle taken in real time by the binocular camera, and the mean of the depth information of multiple key nodes of the pedestrian target on the ROI depth map is calculated by obtaining the ROI depth map after removing the obstructions in front of the pedestrian, as the distance between the current pedestrian and the vehicle, and displayed in real time on the vehicle display screen, that is, through the detection of key nodes of the human body and the extraction of the region of interest, the speed and accuracy of the detection and ranging of pedestrians in front of the vehicle are improved, and when there are obstructions in front of the pedestrian target, the identification of the pedestrian and the measurement of the distance are not hindered, so that the automatic driving vehicle can quickly and accurately measure the distance with the pedestrian in front of the vehicle in a complex traffic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is an overall flow chart of a method for measuring the distance between pedestrians in front of a vehicle based on a binocular camera according to an embodiment of the present invention;

[0018] Figure 2 A specific flow chart of a method for measuring distance between pedestrians in front of a vehicle based on a binocular camera according to an embodiment of the present invention;

[0019] Figure 3The present invention is a schematic structural diagram of a vehicle-front pedestrian ranging terminal based on a binocular camera according to an embodiment of the present invention.

[0020] Description of labels:

[0021] 1. A pedestrian ranging terminal based on a binocular camera; 2. Memory; 3. Processor. DETAILED DESCRIPTION

[0022] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0023] Prior to this, the English abbreviations and professional terms involved in this invention are explained as follows:

[0024] 1. ROI, Region Of Interest, region of interest;

[0025] 2. AlphaPose code, an open source human posture recognition code;

[0026] 3. SGBM algorithm, semi-global block matching, a semi-global matching algorithm used to calculate disparity in binocular vision;

[0027] 4. SAD, Sum of absolute differences, an image matching algorithm;

[0028] 5. Zhang’s calibration method: Dr. Zhang Zhengyou proposed a practical method for camera calibration using a plane checkerboard in his paper “Flexible Camera Calibration By Viewing a Plane From Unknown Orientations” published in the top international conference ICCV in 1999;

[0029] 6. Yolov4 neural network algorithm, a target detection algorithm based on neural network.

[0030] Please refer to Figure 1 and Figure 2 , a method for measuring the distance between pedestrians in front of a vehicle based on a binocular camera, comprising the steps of:

[0031] S1. Calibrate and stereo match the binocular camera deployed on the top of the outer side of the vehicle, and train a pedestrian recognition model based on a set of pedestrian videos pre-taken by the binocular camera;

[0032] S2, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera, if the pedestrian target is recognized, extracting the ROI area of ​​the pedestrian target on the current image in front of the vehicle, and obtaining the ROI depth map after removing the obstruction;

[0033] S3. Randomly extract multiple key nodes of the pedestrian target, and obtain the depth information of each key node in the ROI depth map, use the average of the depth information of each key node as the pedestrian distance, and display the vehicle front image in which the pedestrian target is identified and the pedestrian distance in real time on the vehicle display screen.

[0034] From the above description, it can be seen that the beneficial effects of the present invention are: by deploying a binocular camera at the top of the outer side of the vehicle, calibrating the binocular camera, stereo matching and training the pedestrian recognition model, when the vehicle is in the process of automatic driving, the pedestrian target can be recognized in time by the pedestrian recognition model on the image in front of the vehicle taken in real time by the binocular camera, and by obtaining the pedestrian target ROI depth map after removing the obstructions in front of the pedestrian, the average of the depth information of multiple key nodes of the pedestrian target on the ROI depth map is calculated as the current distance between the pedestrian and the vehicle, and displayed in real time on the on-board display screen, that is, through the detection of key nodes of the human body and the extraction of the area of ​​interest, the rate and accuracy of pedestrian detection and ranging in front of the vehicle are improved, and when there are obstructions in front of the pedestrian target, the identification of the pedestrian and the accurate measurement of the distance are not hindered, which meets the requirements of the automatic driving vehicle to quickly and accurately measure the distance to the pedestrian in front of the vehicle in a complex traffic environment.

[0035] Furthermore, the step S2 is specifically as follows:

[0036] S21, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera frame by frame, if the pedestrian target is recognized, executing steps S22 and S23, otherwise continue to recognize the next frame of the image in front of the vehicle;

[0037] S22, intercepting the minimum circumscribed rectangular area of ​​the pedestrian target on the front image of the vehicle in the current frame, and obtaining the coordinates of four vertices of the minimum circumscribed rectangle to generate an image mask;

[0038] S23, extracting the ROI region where the pedestrian target is located in the image mask, removing background information and information about an obstruction located in front of the pedestrian target, and then performing stereo correction on the ROI region using the stereo matching to obtain a ROI depth map.

[0039] From the above description, it can be seen that the pedestrian recognition model is first used to recognize pedestrian targets in the front image of the vehicle taken in real time by the binocular camera. Only the front image of the vehicle in which the pedestrian target is recognized needs to be subsequently processed, and there is no need to process the front image of the vehicle in which the pedestrian target is not recognized, thereby improving the efficiency of pedestrian recognition in front of the vehicle during driving; at the same time, by intercepting the pedestrian target area of ​​the front image of the vehicle in which the pedestrian target is recognized, the ROI area where the pedestrian is located is obtained, and then the occlusion and background information of the ROI area are removed, and then stereo matching is performed to obtain the ROI depth map of only the pedestrian area, which not only reduces the image processing pressure of the binocular camera, but also avoids the error of subsequent pedestrian distance measurement in front of the vehicle caused by occlusion of the pedestrian in front of the pedestrian, thereby further improving the accuracy of ranging.

[0040] Furthermore, in step S3, a plurality of key nodes of the pedestrian target are randomly extracted, specifically:

[0041] The AlphaPose code is used to randomly extract 18 key nodes of the pedestrian target.

[0042] From the above description, we can see that the 18 key nodes of the recognized pedestrian target are extracted through the human posture recognition code - AlphaPose code, and then the final distance of the pedestrian in front of the vehicle is determined by multiple key nodes, which further improves the accuracy of distance measurement.

[0043] Furthermore, in step S1, the binocular camera is calibrated, specifically:

[0044] The binocular camera is calibrated by using Zhang's calibration method and chessboard to obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the binocular camera;

[0045] In step S1, stereo matching is performed on the binocular cameras, specifically:

[0046] Using the SGBM algorithm, a frame of image is obtained from the pedestrian video set pre-taken by the binocular camera, and the distortion parameter is used to perform distortion correction on the frame of image, and the internal parameter and the external parameter are used to perform stereo correction on the frame of image to obtain a depth map of the corrected image;

[0047] Repeat the steps of adjusting the mismatch rate, SAD window size and disparity window size of the image depth map for multiple times until the error between the depth information of the image depth map and the actual distance between the pedestrian and the binocular camera in the corresponding frame of the certain image frame is less than a preset distance.

[0048] From the above description, it can be seen that Zhang's calibration method is used to calibrate the binocular camera. The calibration process only requires the use of a chessboard and the capture of several sets of images from different directions. It is not only practical, flexible and convenient, but also has high accuracy and good robustness. At the same time, the SGBM algorithm is used to perform distortion correction and stereo correction on the parameters of the calibrated binocular camera to correct the erroneous matching caused by parallax, making the distance measurement accuracy of the binocular camera more accurate.

[0049] Furthermore, in step S1, the pedestrian recognition model is obtained by training based on the pedestrian video set pre-shot by the binocular camera, specifically:

[0050] The yolov4 neural network algorithm is adopted, and the binocular camera is used to pre-take pedestrian images in different scenes during vehicle driving, and the pedestrians in the pedestrian images are marked. The marked pedestrian images are randomly divided into training set, test set and verification set according to a preset ratio. After training, testing and verification, the pedestrian recognition model is finally obtained.

[0051] From the above description, it can be seen that the pedestrian recognition model of the binocular camera is trained through the yolov4 neural network algorithm, and then the pedestrian recognition model is directly used to perform pedestrian recognition on the front image of the vehicle taken in real time by the binocular camera, which is fast and accurate.

[0052] Please refer to Figure 3 , a vehicle-front pedestrian ranging terminal based on a binocular camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0053] S1. Calibrate and stereo match the binocular camera deployed on the top of the outer side of the vehicle, and train a pedestrian recognition model based on a set of pedestrian videos pre-taken by the binocular camera;

[0054] S2, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera, if the pedestrian target is recognized, extracting the ROI area of ​​the pedestrian target on the current image in front of the vehicle, and obtaining the ROI depth map after removing the obstruction;

[0055] S3. Randomly extract multiple key nodes of the pedestrian target, and obtain the depth information of each key node in the ROI depth map, take the average of the depth information of each key node as the pedestrian distance, and display the front image of the identified pedestrian target and the pedestrian distance in real time on the vehicle display screen.

[0056] From the above description, it can be seen that the beneficial effects of the present invention are: based on the same technical concept, in conjunction with the above-mentioned method for measuring the distance between pedestrians in front of a vehicle based on a binocular camera, a terminal for measuring the distance between pedestrians in front of a vehicle based on a binocular camera is provided, by deploying a binocular camera at the top of the outer side of the vehicle, calibrating the binocular camera, stereo matching and training the pedestrian recognition model, when the vehicle is in the process of automatic driving, the pedestrian target can be recognized in time by the pedestrian recognition model for the image in front of the vehicle taken in real time by the binocular camera, and the depth information of multiple key nodes of the pedestrian target on the ROI depth map is calculated by obtaining the ROI depth map after removing the obstruction in front of the pedestrian, and the mean value of the depth information of multiple key nodes of the pedestrian target on the ROI depth map is used as the distance between the current pedestrian and the vehicle, and is displayed in real time on the vehicle display screen, that is, by detecting the key nodes of the human body and extracting the region of interest, the speed and accuracy of detecting and measuring the distance between pedestrians in front of the vehicle are improved, and when there are obstructions in front of the pedestrian target, it is not hindered, so that the automatic driving vehicle can quickly and accurately measure the distance with the pedestrian in front of the vehicle in a complex traffic environment.

[0057] Furthermore, the step S2 is specifically as follows:

[0058] S21, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera frame by frame, if the pedestrian target is recognized, executing steps S22 and S23, otherwise continue to recognize the next frame of the image in front of the vehicle;

[0059] S22, intercepting the minimum circumscribed rectangular area of ​​the pedestrian target on the front image of the vehicle in the current frame, and obtaining the coordinates of four vertices of the minimum circumscribed rectangle to generate an image mask;

[0060] S23, extracting the ROI region where the pedestrian target is located in the image mask, removing background information and information about an obstruction located in front of the pedestrian target, and then performing stereo correction on the ROI region using the stereo matching to obtain a ROI depth map.

[0061] From the above description, it can be seen that the pedestrian recognition model is first used to recognize pedestrian targets in the front image of the vehicle taken in real time by the binocular camera. Only the front image of the vehicle in which the pedestrian target is recognized needs to be subsequently processed, and there is no need to process the front image of the vehicle in which the pedestrian target is not recognized, thereby improving the efficiency of pedestrian recognition in front of the vehicle during driving; at the same time, by intercepting the pedestrian target area of ​​the front image of the vehicle in which the pedestrian target is recognized, the ROI area where the pedestrian is located is obtained, and then the occlusion and background information of the ROI area are removed, and then stereo matching is performed to obtain the ROI depth map of only the pedestrian area, which not only reduces the image processing pressure of the binocular camera, but also avoids the error of subsequent pedestrian distance measurement in front of the vehicle caused by occlusion of the pedestrian in front of the pedestrian, thereby further improving the accuracy of ranging.

[0062] Furthermore, in step S3, a plurality of key nodes of the pedestrian target are randomly extracted, specifically:

[0063] The AlphaPose code is used to randomly extract 18 key nodes of the pedestrian target.

[0064] From the above description, we can see that the 18 key nodes of the recognized pedestrian target are extracted through the human posture recognition code - AlphaPose code, and then the final distance of the pedestrian in front of the vehicle is determined by multiple key nodes, which further improves the accuracy of distance measurement.

[0065] Furthermore, in step S1, the binocular camera is calibrated, specifically:

[0066] The binocular camera is calibrated by using Zhang's calibration method and chessboard to obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the binocular camera;

[0067] In step S1, stereo matching is performed on the binocular cameras, specifically:

[0068] Using the SGBM algorithm, a frame of image is obtained from the pedestrian video set pre-taken by the binocular camera, and the distortion parameter is used to perform distortion correction on the frame of image, and the internal parameter and the external parameter are used to perform stereo correction on the frame of image to obtain a depth map of the corrected image;

[0069] Repeat the steps of adjusting the mismatch rate, SAD window size and disparity window size of the image depth map for multiple times until the error between the depth information of the image depth map and the actual distance between the pedestrian and the binocular camera in the corresponding frame of the certain image frame is less than a preset distance.

[0070] From the above description, it can be seen that Zhang's calibration method is used to calibrate the binocular camera. The calibration process only requires the use of a chessboard and the capture of several sets of images from different directions. It is not only practical, flexible and convenient, but also has high accuracy and good robustness. At the same time, the SGBM algorithm is used to perform distortion correction and stereo correction on the parameters of the calibrated binocular camera to correct the erroneous matching caused by parallax, making the distance measurement accuracy of the binocular camera more accurate.

[0071] Furthermore, in step S1, the pedestrian recognition model is obtained by training based on the pedestrian video set pre-shot by the binocular camera, specifically:

[0072] The yolov4 neural network algorithm is adopted, and the binocular camera is used to pre-take pedestrian images in different scenes during vehicle driving, and the pedestrians in the pedestrian images are marked. The marked pedestrian images are randomly divided into training set, test set and verification set according to a preset ratio. After training, testing and verification, the pedestrian recognition model is finally obtained.

[0073] From the above description, it can be seen that the pedestrian recognition model of the binocular camera is trained through the yolov4 neural network algorithm, and then the pedestrian recognition model is directly used to perform pedestrian recognition on the front image of the vehicle taken in real time by the binocular camera, which is fast and accurate.

[0074] The present invention provides a method and terminal for measuring the distance between pedestrians in front of a vehicle based on a binocular camera, which is suitable for timely identifying and accurately measuring the distance of pedestrians appearing in front of an autonomous driving vehicle during driving, and displaying them to passengers so that they can take corresponding actions in a timely manner. The method and terminal are described in detail in conjunction with an embodiment below.

[0075] Please refer to Figure 1 , Embodiment 1 of the present invention is:

[0076] A method for measuring the distance between pedestrians in front of a vehicle based on a binocular camera, such as Figure 1 As shown, the steps include:

[0077] S1. Calibrate and stereo match the binocular camera deployed on the top of the outer side of the vehicle, and train a pedestrian recognition model based on a set of pedestrian videos pre-taken by the binocular camera.

[0078] S2, start the binocular camera, use the pedestrian recognition model to identify pedestrian targets in the image in front of the vehicle taken in real time by the binocular camera, if the pedestrian target is identified, extract the ROI area of ​​the pedestrian target on the current image in front of the vehicle, and obtain the ROI depth map after removing the obstruction.

[0079] S3. Randomly extract multiple key nodes of pedestrian targets, and obtain the depth information of each key node in the ROI depth map, use the average of the depth information of each key node as the pedestrian distance, and display the front image of the vehicle where the pedestrian target is identified and the pedestrian distance in real time on the vehicle display screen.

[0080] That is, in this embodiment, firstly, a binocular camera can be deployed on the top of the outer side of the vehicle, and the position and angle of the binocular camera can be adjusted so that the binocular camera can clearly capture the scene in front of the vehicle. Then, the fixed binocular camera can be calibrated, stereo matched, and trained with a pedestrian recognition model. When the vehicle is in the process of automatic driving, the pedestrian target can be recognized in time by the pedestrian recognition model on the image in front of the vehicle taken in real time by the binocular camera. The mean of the depth information of multiple key nodes of the pedestrian target on the ROI depth map is calculated by obtaining the ROI depth map after removing the obstructions in front of the pedestrian, as the current distance between the pedestrian and the vehicle, and displayed in real time on the on-board display screen. That is, the detection of key nodes of the human body and the extraction of the region of interest improve the speed and accuracy of pedestrian detection and ranging in front of the vehicle. Moreover, when there are obstructions in front of the pedestrian target, the identification of the pedestrian and the accurate measurement of the distance are not hindered, which satisfies the automatic driving vehicle to quickly and accurately measure the distance to the pedestrian in front of the vehicle in a complex traffic environment. In this embodiment, a central processing controller, such as a Raspberry Pi, can be set up inside the vehicle, and a gigabit network cable can be used to connect the Raspberry Pi, the vehicle display screen and the binocular camera to the same network to achieve communication and transmission between the devices. For example, the binocular camera transmits the captured images to the Raspberry Pi, and the training of the pedestrian recognition model, the calibration of the binocular camera and the stereo matching process can also be implemented in the Raspberry Pi. Then, the pedestrian recognition model is used in the Raspberry Pi to perform pedestrian recognition and pedestrian distance measurement on the front image of the vehicle captured in real time by the binocular camera, and then the image is transmitted to the vehicle display screen for display.

[0081] Please refer to Figure 2 , Embodiment 2 of the present invention is:

[0082] A method for measuring the distance between pedestrians in front of a vehicle based on a binocular camera. Based on the above-mentioned embodiment 1, in this embodiment, the binocular camera is calibrated in step S1, specifically:

[0083] The binocular camera is calibrated using Zhang's calibration method and chessboard to obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the binocular camera. In this embodiment, the intrinsic parameters, extrinsic parameters and distortion parameters of the binocular camera are obtained as follows: intrinsic parameter matrix (f, f / dx, f / dy, cx, cy), distortion parameters (k1, k2, k3, k4, k5) and extrinsic parameter matrix (R, t), where f is the focal length, f / dx and f / dy are the normalized focal lengths on the x-axis and y-axis, cx and cy are the centers of the image, k1, k2 and k3 are radial distortion parameters, k4 and k5 are tangential distortion parameters, R is the rotation matrix, and t is the translation vector.

[0084] That is, in this embodiment, the binocular camera is calibrated by using Zhang's calibration method. The calibration process only needs to use a chessboard and take several groups of images from different directions. It is not only practical, flexible and convenient, but also has high accuracy and good robustness.

[0085] Among them, in step S1, stereo matching is performed on the binocular cameras, specifically:

[0086] The SGBM algorithm is used to obtain a frame of image from the pedestrian video set pre-taken by the binocular camera, and the distortion parameters are used to correct the distortion of the frame of image, and the internal and external parameters are used to perform stereo correction on the frame of image to obtain the depth map of the corrected image.

[0087] The three parameters of the image depth map, namely, the mismatch rate, the SAD window size, and the disparity window size, are repeatedly adjusted until the error between the depth information of the image depth map and the actual distance between the pedestrian and the binocular camera in a certain frame of the image is less than the preset distance.

[0088] That is, in this embodiment, the SGBM algorithm is used to perform distortion correction and stereo correction on the parameters of the calibrated binocular camera at the same time to correct the mismatch caused by parallax, so that the distance measurement accuracy of the binocular camera is more accurate. In this embodiment, the preset distance in step S22 can be set to 0.1m.

[0089] In this embodiment, the pedestrian recognition model is obtained by training based on the pedestrian video set pre-shot by the binocular camera in step S1, specifically:

[0090] The yolov4 neural network algorithm is adopted, and the binocular camera is used to pre-take pedestrian images in different scenes during vehicle driving. The pedestrians in the pedestrian images are marked, and the marked pedestrian images are randomly divided into training set, test set and verification set according to the preset ratio. After training, testing and verification, the pedestrian recognition model is finally obtained.

[0091] That is, the pedestrian recognition model of the binocular camera is trained through the yolov4 neural network algorithm, and then the pedestrian recognition model is directly used to perform pedestrian recognition on the front image of the vehicle taken in real time by the binocular camera, which is fast and accurate.

[0092] In this embodiment, step S2 is specifically as follows:

[0093] S21, start the binocular camera, and use the pedestrian recognition model to perform pedestrian target recognition frame by frame on the front image of the vehicle taken in real time by the binocular camera. If a pedestrian target is recognized, execute steps S22 and S23, otherwise continue to recognize the next frame of the front image of the vehicle.

[0094] S22, intercepting the minimum circumscribed rectangular area of ​​the pedestrian target on the front image of the vehicle in the current frame, and obtaining the coordinates of the four vertices of the minimum circumscribed rectangle to generate an image mask.

[0095] S23, extracting the ROI region where the pedestrian target is located in the image mask, removing background information and information about obstructions located in front of the pedestrian target, and then using stereo matching to perform stereo correction on the ROI region to obtain a ROI depth map.

[0096] That is, in this embodiment, the pedestrian recognition model is first used to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera. Only the image in front of the vehicle in which the pedestrian target is recognized needs to be subsequently processed, and there is no need to process the image in front of the vehicle in which the target pedestrian is not recognized, thereby improving the efficiency of pedestrian recognition in front of the vehicle during driving; at the same time, by intercepting the pedestrian target area of ​​the image in front of the vehicle in which the pedestrian target is recognized, the ROI area where the pedestrian is located is obtained, and then the occlusion and background information of the ROI area are removed, and then stereo matching is performed to obtain the ROI depth map of only the pedestrian area, which not only reduces the image processing pressure of the binocular camera, but also avoids the error in the subsequent distance measurement of the pedestrian in front of the vehicle caused by the occlusion of the pedestrian in front of the pedestrian, thereby further improving the accuracy of ranging.

[0097] Then, after obtaining the ROI depth map, it is necessary to obtain the key nodes of the pedestrian target, that is, randomly extract multiple key nodes of the pedestrian target in step S3, which are specifically:

[0098] The AlphaPose code is used to randomly extract 18 key nodes of pedestrian targets.

[0099] That is, the 18 key nodes of the identified pedestrian target are extracted through the human posture recognition code - AlphaPose code, and the final distance of the pedestrian in front of the vehicle is determined by the average of the depth information of multiple key nodes, further improving the accuracy of distance measurement. In this embodiment, the AlphaPose code can be pre-deployed in the central processor inside the vehicle.

[0100] Please refer to Figure 3 , Embodiment 3 of the present invention is:

[0101] A vehicle-front pedestrian ranging terminal 1 based on a binocular camera comprises a memory 2, a processor 3 and a computer program stored in the memory 2 and executable on the processor 3. In this embodiment, the processor 3 implements the steps of any one of the above-mentioned embodiments 1 to 2 when executing the computer program.

[0102] In summary, the present invention provides a method and terminal for measuring the distance between pedestrians in front of a vehicle based on a binocular camera. A binocular camera is deployed on the top of the outer side of the vehicle, and the binocular camera is calibrated, stereo matched, and a pedestrian recognition model is trained. When the vehicle is in the process of automatic driving, the pedestrian target can be recognized in time by the pedestrian recognition model for the image in front of the vehicle taken in real time by the binocular camera, and the ROI depth map of the pedestrian target after removing the obstruction in front of the pedestrian is obtained. The mean of the depth information of multiple key nodes of the pedestrian target on the ROI depth map is calculated as the current distance between the pedestrian and the vehicle, and is displayed in real time on the on-board display screen. Even if there is an obstruction in front of the vehicle, it does not hinder the recognition of pedestrians and the measurement of the distance, thereby meeting the requirements of the automatic driving vehicle to quickly and accurately measure the distance to the pedestrian in front of the vehicle in a complex traffic environment.

[0103] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for measuring the distance between pedestrians in front of a vehicle based on a binocular camera, characterized in that: Includes steps: S1. Calibrate and stereo match the binocular camera deployed on the top of the outer side of the vehicle, and train a pedestrian recognition model based on a set of pedestrian videos pre-taken by the binocular camera; S2, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera, if the pedestrian target is recognized, extracting the ROI area of ​​the pedestrian target on the current image in front of the vehicle, and obtaining the ROI depth map after removing the obstruction; S3, randomly extracting multiple key nodes of the pedestrian target, and obtaining the depth information of each key node in the ROI depth map, taking the average of the depth information of each key node as the pedestrian distance, and displaying the front image of the vehicle in which the pedestrian target is identified and the pedestrian distance in real time on the vehicle display screen; The step S2 is specifically as follows: S21, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera frame by frame, if the pedestrian target is recognized, executing steps S22 and S23, otherwise continue to recognize the next frame of the image in front of the vehicle; S22, intercepting the minimum circumscribed rectangular area of ​​the pedestrian target on the front image of the vehicle in the current frame, and obtaining the coordinates of four vertices of the minimum circumscribed rectangle to generate an image mask; S23, extracting the ROI region where the pedestrian target is located in the image mask, removing background information and information about an obstruction located in front of the pedestrian target, and then performing stereo correction on the ROI region using the stereo matching to obtain a ROI depth map.

2. The method for measuring the distance between pedestrians in front of a vehicle based on a binocular camera according to claim 1, characterized in that: In step S3, a plurality of key nodes of the pedestrian target are randomly extracted, specifically: The AlphaPose code is used to randomly extract 18 key nodes of the pedestrian target.

3. The method for measuring the distance between pedestrians in front of a vehicle and a binocular camera according to claim 1, characterized in that: In step S1, the binocular camera is calibrated, specifically: The binocular camera is calibrated by using Zhang's calibration method and chessboard to obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the binocular camera; In step S1, stereo matching is performed on the binocular cameras, specifically: Using the SGBM algorithm, a frame of image is obtained from the pedestrian video set pre-taken by the binocular camera, and the distortion parameter is used to perform distortion correction on the frame of image, and the internal parameter and the external parameter are used to perform stereo correction on the frame of image to obtain a depth map of the corrected image; Repeat the steps of adjusting the mismatch rate, SAD window size and disparity window size of the image depth map for multiple times until the error between the depth information of the image depth map and the actual distance between the pedestrian and the binocular camera in the corresponding frame of the certain image frame is less than a preset distance.

4. The method for measuring the distance between pedestrians in front of a vehicle and a vehicle based on a binocular camera according to claim 3, characterized in that: In step S1, a pedestrian recognition model is obtained by training based on a set of pedestrian videos pre-taken by the binocular camera, specifically: The yolov4 neural network algorithm is adopted, and the binocular camera is used to pre-take pedestrian images in different scenes during vehicle driving, and the pedestrians in the pedestrian images are marked. The marked pedestrian images are randomly divided into training set, test set and verification set according to a preset ratio. After training, testing and verification, the pedestrian recognition model is finally obtained.

5. A vehicle-front pedestrian ranging terminal based on a binocular camera, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: S1. Calibrate and stereo match the binocular camera deployed on the top of the outer side of the vehicle, and train a pedestrian recognition model based on a set of pedestrian videos pre-taken by the binocular camera; S2, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera, if the pedestrian target is recognized, extracting the ROI area of ​​the pedestrian target on the current image in front of the vehicle, and obtaining the ROI depth map after removing the obstruction; S3, randomly extracting multiple key nodes of the pedestrian target, and obtaining the depth information of each key node in the ROI depth map, taking the average of the depth information of each key node as the pedestrian distance, and displaying the front image of the vehicle in which the pedestrian target is identified and the pedestrian distance in real time on the vehicle display screen; The step S2 is specifically as follows: S21, starting the binocular camera, using the pedestrian recognition model to perform pedestrian target recognition on the image in front of the vehicle taken in real time by the binocular camera frame by frame, if the pedestrian target is recognized, executing steps S22 and S23, otherwise continue to recognize the next frame of the image in front of the vehicle; S22, intercepting the minimum circumscribed rectangular area of ​​the pedestrian target on the front image of the vehicle in the current frame, and obtaining the coordinates of four vertices of the minimum circumscribed rectangle to generate an image mask; S23, extracting the ROI region where the pedestrian target is located in the image mask, removing background information and information about an obstruction located in front of the pedestrian target, and then performing stereo correction on the ROI region using the stereo matching to obtain a ROI depth map.

6. The vehicle-front pedestrian ranging terminal based on a binocular camera according to claim 5, characterized in that: In step S3, a plurality of key nodes of the pedestrian target are randomly extracted, specifically: The AlphaPose code is used to randomly extract 18 key nodes of the pedestrian target.

7. The vehicle-front pedestrian ranging terminal based on a binocular camera according to claim 5, characterized in that: In step S1, the binocular camera is calibrated, specifically: The binocular camera is calibrated by using Zhang's calibration method and chessboard to obtain the intrinsic parameters, extrinsic parameters and distortion parameters of the binocular camera; In step S1, stereo matching is performed on the binocular cameras, specifically: Using the SGBM algorithm, a frame of image is obtained from the pedestrian video set pre-taken by the binocular camera, and the distortion parameter is used to perform distortion correction on the frame of image, and the internal parameter and the external parameter are used to perform stereo correction on the frame of image to obtain a depth map of the corrected image; Repeat the steps of adjusting the mismatch rate, SAD window size and disparity window size of the image depth map for multiple times until the error between the depth information of the image depth map and the actual distance between the pedestrian and the binocular camera in the corresponding frame of the certain image frame is less than a preset distance.

8. The vehicle-front pedestrian ranging terminal based on binocular camera according to claim 7, characterized in that: In step S1, a pedestrian recognition model is obtained by training based on a set of pedestrian videos pre-taken by the binocular camera, specifically: The yolov4 neural network algorithm is adopted, and the binocular camera is used to pre-take pedestrian images in different scenes during vehicle driving, and the pedestrians in the pedestrian images are marked. The marked pedestrian images are randomly divided into training set, test set and verification set according to a preset ratio. After training, testing and verification, the pedestrian recognition model is finally obtained.

Images