A pedestrian detection method and system based on computer vision

Abstract

The present invention is applicable to the technical field of pedestrian detection, and particularly relates to a pedestrian detection method and system based on computer vision. The method includes: constructing a basic intersection model; collecting images through a binocular camera, calculating distances, determining the positions of various objects in the picture, and generating a standard intersection model; delimiting a pedestrian detection area, identifying pedestrians located in the pedestrian detection area through the binocular camera, marking each pedestrian, and calculating the motion prediction trajectories of each pedestrian; identifying each vehicle, calculating the motion prediction trajectory of the vehicle, performing line-of-sight occlusion verification and trajectory verification, and issuing a warning message based on the verification results. The present invention uses a binocular camera to determine the driving trajectories of pedestrians and vehicles. When there is line-of-sight occlusion, the vehicle is warned through a warning message, avoiding traffic accidents caused by line-of-sight occlusion and improving traffic safety.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian detection, and particularly relates to a pedestrian detection method and system based on computer vision. Background Art

[0002] Pedestrian detection, as a key part of computer vision technology, aims to automatically identify and accurately locate pedestrians from images or videos.

[0003] During the traffic management process, warning devices are often set at intersections, which can warn passing pedestrians when the red light is on. The existing warning devices can complete the warning of pedestrians, but they cannot prompt the driver when there is a visual obstacle. Therefore, when pedestrians fail to cross the crosswalk within the time period when the green light is on, the starting and passing of vehicles will pose a threat to pedestrians. Summary of the Invention

[0004] The purpose of the present invention is to provide a pedestrian detection method based on computer vision, aiming to solve the problem that when pedestrians fail to cross the crosswalk within the time period when the green light is on, the starting and passing of vehicles will pose a threat to pedestrians.

[0005] The present invention is implemented as follows. A pedestrian detection method based on computer vision, the method includes:

[0006] Construct a basic intersection model, the basic intersection model at least includes a road model, a building model, and a device model, and the device model at least includes a model of a binocular camera;

[0007] Collect images through a binocular camera, calculate the distance of the collected binocular ranging images, determine the positions of various objects in the picture, and generate a standard intersection model;

[0008] Define a pedestrian detection area in the standard intersection model, identify pedestrians located in the pedestrian detection area through a binocular camera, mark each pedestrian, and calculate the motion prediction trajectory of each pedestrian;

[0009] Identify each vehicle through a binocular camera, calculate the motion prediction trajectory of the vehicle, perform line-of-sight occlusion checking and trajectory checking based on the standard intersection model, and send a warning message based on the checking result.

[0010] Preferably, the step of collecting images through a binocular camera, calculating the distance of the collected binocular ranging images, determining the positions of various objects in the picture, and generating a standard intersection model specifically includes:

[0011] Collect a to-be-calculated image through a binocular camera, and identify various objects in the to-be-calculated image;

[0012] Calculate the reference distance values between each object and the fixed reference objects in the intersection using the binocular ranging algorithm;

[0013] Retrieve the basic intersection model, and generate the dynamic models of each object in the basic intersection model based on the reference distance values to obtain the standard intersection model.

[0014] Preferably, the steps of demarcating a pedestrian detection area in the standard intersection model, identifying pedestrians located within the pedestrian detection area through a binocular camera, marking each pedestrian, and calculating the motion prediction trajectory of each pedestrian specifically include:

[0015] Based on the passing direction of the pedestrians, set a pedestrian detection area in the standard intersection model, and the pedestrian detection area covers the crosswalk;

[0016] Collect images at a preset acquisition time interval to obtain the binocular images to be detected, and the binocular images to be detected include two sets of actual captured images, and the two sets of actual captured images come from two cameras in the binocular camera respectively;

[0017] Use the face recognition algorithm to identify the pedestrians in the picture, mark each pedestrian, determine the motion trajectory of the pedestrians according to the consecutive binocular images to be detected, generate the motion prediction trajectory, and generate pedestrian information.

[0018] Preferably, the steps of identifying each vehicle through a binocular camera, calculating the motion prediction trajectory of the vehicle, performing line-of-sight occlusion checking and trajectory checking based on the standard intersection model, and sending a warning message based on the checking result specifically include:

[0019] Retrieve the binocular images to be detected collected by the binocular camera, identify the vehicles included therein, and generate vehicle information, and the vehicle information includes at least vehicle type, vehicle size, and driver's seat height;

[0020] Based on the vehicle information and pedestrian information, determine the positions of the pedestrians and vehicles in the standard intersection model, and determine the motion trajectories of the pedestrians and vehicles, complete the line-of-sight occlusion checking and trajectory checking, and obtain the checking result;

[0021] When it is determined based on the checking result that there is a line-of-sight occlusion between the vehicle and the pedestrian and their trajectories overlap, a warning message is sent.

[0022] Preferably, the step of sending the warning message includes sending a control signal to the traffic lights to warn the vehicles through the traffic lights.

[0023] Another object of the present invention is to provide a pedestrian detection system based on computer vision, and the system includes:

[0024] The basic model construction module is used to construct the basic intersection model, where the basic intersection model at least includes a road model, a building model, and a device model, and the device model at least includes the model of a binocular camera;

[0025] The standard model generation module is used to collect images through a binocular camera, calculate the distances of the collected binocular ranging images, determine the positions of various objects in the image, and generate the intersection standard model;

[0026] The pedestrian trajectory prediction module is used to delimit the pedestrian detection area in the intersection standard model, identify the pedestrians located in the pedestrian detection area through a binocular camera, mark each pedestrian, and calculate the motion prediction trajectory of each pedestrian;

[0027] The safety verification module is used to identify each vehicle through a binocular camera, calculate the motion prediction trajectory of the vehicle, perform line-of-sight occlusion verification and trajectory verification based on the intersection standard model, and send out a warning message based on the verification result.

[0028] Preferably, the standard model generation module includes:

[0029] The object recognition unit is used to collect an image to be measured through a binocular camera and identify various objects in the image to be measured;

[0030] The distance value calculation unit is used to calculate the reference distance value between each object and the fixed reference object in the intersection by using the binocular ranging algorithm;

[0031] The standard model generation unit is used to retrieve the basic intersection model and generate the dynamic model of each object in the basic intersection model based on the reference distance value to obtain the intersection standard model.

[0032] Preferably, the pedestrian trajectory prediction module includes:

[0033] The area division unit is used to set the pedestrian detection area in the intersection standard model based on the pedestrian's passing direction, and the pedestrian detection area covers the crosswalk;

[0034] The image acquisition unit is used to collect images at a preset acquisition time interval to obtain the binocular images to be detected, where the binocular images to be detected include two sets of actual captured images, and the two sets of actual captured images come from the two cameras in the binocular camera respectively;

[0035] The pedestrian recognition unit is used to identify the pedestrians in the image by using the face recognition algorithm, mark each pedestrian, determine the motion trajectory of the pedestrian according to the continuous binocular images to be detected, generate the motion prediction trajectory, and generate the pedestrian information.

[0036] Preferably, the safety verification module includes:

[0037] A vehicle recognition unit is configured to retrieve the to-be-detected binocular images collected by a binocular camera, recognize the vehicles included therein, and generate vehicle information, where the vehicle information at least includes vehicle type, vehicle size, and driver's seat height;

[0038] A trajectory verification unit is configured to determine the positions of pedestrians and vehicles in the intersection standard model based on the vehicle information and pedestrian information, and determine the movement trajectories of pedestrians and vehicles, complete line-of-sight occlusion verification and trajectory verification, and obtain a verification result;

[0039] A risk warning unit is configured to issue a warning message when it is determined based on the verification result that there is a line-of-sight occlusion between the vehicle and the pedestrian and their trajectories overlap.

[0040] Preferably, the step of issuing the warning message includes sending a control signal to the traffic light to warn the vehicle through the traffic light.

[0041] A pedestrian detection method based on computer vision provided by the present invention can determine the distribution of the basic building structures at the intersection by constructing an intersection standard model, and then use a binocular camera to determine the driving trajectories of pedestrians and vehicles. When there is a line-of-sight occlusion, the vehicle is warned through a warning message, avoiding traffic accidents caused by line-of-sight occlusion and improving traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of a pedestrian detection method based on computer vision provided by an embodiment of the present invention;

[0043] Figure 2 It is a flowchart of the steps of performing image acquisition through a binocular camera, calculating the distances of the collected binocular ranging images, determining the positions of each object in the picture, and generating an intersection standard model;

[0044] Figure 3 It is a flowchart of the steps of demarcating a pedestrian detection area in the intersection standard model, recognizing pedestrians located in the pedestrian detection area through a binocular camera, marking each pedestrian, and calculating the movement prediction trajectories of each pedestrian;

[0045] Figure 4 It is a flowchart of the steps of recognizing each vehicle through a binocular camera, calculating the movement prediction trajectory of the vehicle, performing line-of-sight occlusion verification and trajectory verification based on the intersection standard model, and issuing a warning message based on the verification result;

[0046] Figure 5 It is an architecture diagram of a pedestrian detection system based on computer vision provided by an embodiment of the present invention;

[0047] Figure 6 It is an architecture diagram of a standard model generation module provided by an embodiment of the present invention;

[0048] Figure 7 It is an architecture diagram of a pedestrian trajectory prediction module provided by an embodiment of the present invention;

[0049] Figure 8 It is an architecture diagram of a safety check module provided by an embodiment of the present invention. Detailed implementation manners

[0050] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] It can be understood that the terms "first", "second", etc. used in the present application can be used in this document to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of the present application, the first xx script can be called the second xx script, and similarly, the second xx script can be called the first xx script.

[0052] As Figure 1 shown, it is a flowchart of a pedestrian detection method based on computer vision provided by an embodiment of the present invention, and the method includes:

[0053] S100. Build a basic intersection model, where the basic intersection model at least includes a road model, a building model, and a device model, and the device model at least includes a model of a binocular camera.

[0054] In this step, build a basic intersection model and perform three-dimensional modeling based on the on-site building structure. When performing modeling, it at least includes modeling of the road, modeling of the building, and modeling of the device. For example, for an intersection with an underpass road, there are often fences on the road surface, and the U-shaped area formed by the fences will block the line of sight of vehicles. When pedestrians pass by, it is easy to have the situation of "ghost probes". By setting a binocular camera at the intersection and using the binocular camera to monitor pedestrians and vehicles to ensure traffic safety. During the modeling process, not only the position of the road and the lane distribution of the road need to be recorded in the model, but also the building model needs to be recorded, including the U-shaped fence. Secondly, the basic intersection model adopts high-precision modeling, and the modeling ratio is 1:1. Determine the spatial position and shooting direction of the binocular camera in the basic intersection model according to the position of the binocular camera.

[0055] S200: Collect images through a binocular camera, calculate the distances of the collected binocular ranging images, determine the positions of various objects in the image, and generate a standard intersection model.

[0056] In this step, images are collected through a binocular camera. After the intersection is put into use, there may be many obstacles at the intersection, such as temporarily added guardrails and parked malfunctioning vehicles. Multiple images are collected through the binocular camera. Through image recognition technology, the objects included in the images are identified. Secondly, the binocular ranging technology is used to measure the distances between various objects and the binocular camera. Through multiple image recognitions, if an object is in the same position during multiple image recognition processes, it is recorded in the standard intersection model. Moreover, the size information of various objects, such as height and width, can be calculated using the binocular camera, so as to record the above objects in the standard intersection model and obtain a complete standard intersection model. The standard intersection model is updated at regular intervals, such as once a minute.

[0057] S300: Define a pedestrian detection area in the standard intersection model, identify the pedestrians located in the pedestrian detection area through a binocular camera, mark each pedestrian, and calculate the predicted movement trajectories of each pedestrian.

[0058] In this step, a pedestrian detection area is defined in the standard intersection model. There are zebra crossings at the intersection. Based on the position of the zebra crossing, the pedestrian detection area for pedestrians to pass through is determined in the standard intersection model. To improve the detection accuracy, the range of the pedestrian detection area can be appropriately expanded, that is, the range of the pedestrian detection area includes at least the range where the crosswalk is located. The binocular camera is used to detect the pedestrians located in the pedestrian detection area. During this process, face recognition technology is used to identify the pedestrians and mark them until the pedestrian disappears from the image. The pedestrians are identified at preset time intervals to determine the movement trajectories of the pedestrians in the standard intersection model, and the predicted movement trajectories are generated based on the historical movement trajectories. The predicted movement trajectory is the predicted movement trajectory of the pedestrian in the future time period.

[0059] S400: Identify each vehicle through a binocular camera, calculate the predicted movement trajectory of the vehicle, perform line-of-sight occlusion check and trajectory check based on the standard intersection model, and issue a warning message based on the check results.

[0060] In this step, each vehicle is identified through a binocular camera. Similarly, the images collected by the binocular camera also contain vehicle information. The vehicle at the front of each lane is identified, and the binocular ranging technology is used to determine the accurate position of the vehicle in the intersection standard model. Based on the historical trajectory of the vehicle, the motion prediction trajectory of the vehicle is calculated. Then, based on the motion prediction trajectories of the vehicle and the pedestrian, trajectory verification can be completed to determine whether there will be overlapping points between the two sets of trajectories. And during this process, line-of-sight occlusion verification is carried out. Since there are obstacles in the intersection, such as U-shaped enclosures, pedestrians and vehicles appear on the two right-angled sides of the U-shaped enclosure respectively. The right-angled sides of the U-shaped enclosure or other obstacles may block the pedestrians. Therefore, according to the positions of the vehicle, the pedestrian, and the obstacle, it is determined whether the pedestrian will be blocked by the obstacle, and a verification result is generated. When the verification result shows that the pedestrian will be blocked, a warning message is generated, and the vehicle and the pedestrian are warned through sound and light warnings to improve road traffic safety.

[0061] As Figure 2 shown, as a preferred embodiment of the present invention, the steps of collecting images through a binocular camera, calculating the distance of the collected binocular ranging images, determining the positions of various objects in the picture, and generating an intersection standard model specifically include:

[0062] S201, collect a to-be-measured image through a binocular camera and identify various objects in the to-be-measured image.

[0063] In this step, a to-be-measured image is collected through a binocular camera. When detecting, the to-be-measured image is collected at a preset time interval. Specifically, multiple groups of to-be-measured images can be collected within one minute, such as 10 groups of to-be-measured images. The objects included in the image are identified through an image recognition algorithm, and the types of the objects are identified.

[0064] S202, use the binocular ranging algorithm to calculate the reference distance values between various objects and the fixed reference objects in the intersection.

[0065] S203, retrieve the intersection basic model, and generate dynamic models of various objects in the intersection basic model based on the reference distance values to obtain the intersection standard model.

[0066] In this step, the distance between the object and the binocular camera is calculated using the binocular ranging algorithm. Using the binocular ranging technology, the size information of various objects in the picture can be determined, and in the intersection standard model, the specific positions of various object models are recorded, such as the specific position of the U-shaped enclosure. Therefore, the positions of various objects, vehicles, and pedestrians in the intersection standard model can be determined.

[0067] As Figure 3As shown, as a preferred embodiment of the present invention, the steps of demarcating a pedestrian detection area in the intersection standard model, identifying pedestrians located in the pedestrian detection area through a binocular camera, marking each pedestrian, and calculating the motion prediction trajectory of each pedestrian specifically include:

[0068] S301, Based on the passing direction of pedestrians, set a pedestrian detection area in the intersection standard model, and the pedestrian detection area covers the crosswalk.

[0069] In this step, based on the passing direction of pedestrians, a pedestrian detection area is set in the intersection standard model. In the road, a crosswalk will be set in any passing direction, and accordingly, a pedestrian detection area is set in the intersection standard model.

[0070] S302, Perform image acquisition at a preset acquisition time interval to obtain a binocular image to be detected. The binocular image to be detected includes two sets of actual captured images, and the two sets of actual captured images are respectively from two cameras in the binocular camera.

[0071] In this step, perform image acquisition at a preset acquisition time interval. For example, perform image acquisition once every 50 ms to obtain a set of binocular images to be detected. The binocular camera actually includes two cameras. Therefore, during image acquisition, two sets of actual captured images will be obtained in one shot, that is, one camera generates one set of actual captured images.

[0072] S303, Use a face recognition algorithm to identify pedestrians in the image, mark each pedestrian, determine the motion trajectory of the pedestrians according to consecutive binocular images to be detected, generate a motion prediction trajectory, and generate pedestrian information.

[0073] In this step, a face recognition algorithm is used to identify pedestrians in the image, and the positions of the pedestrians in the image are determined. In order to distinguish between pedestrians, a label is set for each pedestrian. Specifically, it can be a code, such as pedestrian No. A00001. When a pedestrian appears in the image, the pedestrian is always labeled with this code. When the pedestrian no longer appears in the image, a new code is assigned to it when it enters the image again. The binocular images to be detected are collected at time intervals. Therefore, each set of binocular images to be detected will determine a position of each pedestrian in the intersection standard model once. Then, in the intersection standard model, the historical movement trajectory of the pedestrian can be determined. Based on the movement trend of the historical movement trajectory, the movement prediction trajectory of the pedestrian in the short term can be determined. Specifically, a two-dimensional coordinate system can be constructed based on the ground of the intersection standard model, so as to mark the historical positions of the pedestrians in the two-dimensional coordinate system. According to the coordinate points corresponding to the above historical positions (the abscissa is the time value, and the ordinate is the distance value of the pedestrian from one end of the zebra crossing), function fitting is performed to obtain the pedestrian movement trajectory function. Based on the above pedestrian movement trajectory function, the movement prediction trajectory of the pedestrian in the short term can be determined, and pedestrian information is generated. The pedestrian information at least includes the height of the pedestrian and the position information of the pedestrian in the horizontal plane.

[0074] As Figure 4 shown, as a preferred embodiment of the present invention, the step of identifying each vehicle through a binocular camera, calculating the movement prediction trajectory of the vehicle, performing line-of-sight occlusion check and trajectory check based on the intersection standard model, and sending a warning message based on the check result specifically includes:

[0075] S401, retrieve the binocular images to be detected collected by the binocular camera, identify the vehicles contained therein, and generate vehicle information. The vehicle information at least includes vehicle type, vehicle size, and driver's seat height.

[0076] In this step, the binocular images to be detected captured by the binocular camera are retrieved. In the binocular images to be detected, not only pedestrians are included, but also vehicles in each lane. Similarly, using image recognition technology, the license plate numbers, vehicle sizes, and vehicle types of each vehicle in the image are recognized. The binocular camera can calculate the depth information of the objects in the image, that is, the distance from the camera, by simulating the principle of stereoscopic vision of the human eye and capturing images of the same scene from different positions with two cameras. After obtaining the depth information of the object and the vertical height from the camera to the ground, combined with geometric relationships, the size information such as the height or width of the object or task in the image can be calculated. Taking the calculation of the human height as an example, to obtain the depth information: First, match the left and right images captured by the binocular camera, and use the triangulation method to calculate the depth of the object corresponding to each pixel point in the image (i.e., the Z-axis distance); determine the camera height: obtain the accurate height of the binocular camera relative to the ground; measure the pixel coordinates of the top and feet of the person: identify and mark the positions of the top and feet of the person in the image, and obtain the pixel coordinates of these two points in the image; calculate the actual height: using the principle of similar triangles, the human height (H) can be estimated through the following formula:

[0077]

[0078] where h 0 is the height of the binocular camera minus the vertical distance from the bottom of the camera to the ground, h p is the pixel difference from the feet to the top of the person in the image, and l s is the pixel distance corresponding to the depth value from the camera to the plane where the person stands. Based on this, the height of the person can be calculated. Based on the same principle, the width of the person, the height and width of the vehicle can be calculated. Based on the position of the face in the image, the position of the driver's seat can be determined, so as to obtain vehicle information and pedestrian information.

[0079] S402. Based on the vehicle information and pedestrian information, determine the positions of the pedestrians and vehicles in the intersection standard model, and determine the movement trajectories of the pedestrians and vehicles, complete the line-of-sight occlusion check and trajectory check, and obtain the check result.

[0080] S403. When it is determined based on the check result that there is a line-of-sight occlusion between the vehicle and the pedestrian and their trajectories overlap, a warning message is issued.

[0081] In this step, based on the vehicle information and pedestrian information, determine the positions of the pedestrian and the vehicle in the intersection standard model. Specifically, since the intersection standard model is a three-dimensional model, a three-dimensional coordinate system can be constructed based on this intersection standard model. And the predicted movement trajectories of the pedestrian and the vehicle have been determined. First, perform a trajectory coincidence judgment. Draw the two trajectories in the three-dimensional coordinate system to determine the positions of the pedestrian and the vehicle at each moment. If at a certain moment, the distance between the positions of the pedestrian and the vehicle is less than the preset value, it is determined that there is a trajectory coincidence. Subsequently, start the line-of-sight occlusion check. Based on the vehicle information, determine the position of the driver's seat and its coordinates. The steps include: Image acquisition: The binocular camera simultaneously captures the same scene from slightly different perspectives; Disparity calculation: By comparing the pixel position differences (disparities) of the same object in the images captured by the two cameras, the distance of the object from the camera can be known. The greater the disparity, the closer the object is to the camera; the smaller the disparity, the farther the object is; Three-dimensional reconstruction: Using the disparity information and the known baseline distance between the cameras (the distance between the centers of the two cameras), combined with the internal parameters of the cameras (such as focal length) and external parameters (such as the position and orientation of the cameras), calculate the coordinates of the object corresponding to each pixel in the image in the three-dimensional space through triangulation; Depth information and position calculation: Once the depth information of each pixel is obtained, the three-dimensional structure of the entire scene can be reconstructed, including the spatial positions (X, Y, Z coordinates) of the objects. In the same way, determine the coordinates of the pedestrian. To ensure reliability, use the coordinates at half of the pedestrian's height as the coordinates of the pedestrian. Connect the line between the pedestrian coordinates and the driver's seat coordinates. If the connection line between the two is blocked by an obstacle, it is determined that there is a line-of-sight obstacle; otherwise, it is determined that there is no line-of-sight obstacle. When it is determined that there are both line-of-sight obstacles and trajectory overlaps, give an alarm. The alarm method can be to give a warning to the vehicle and the pedestrian through sound and light signals, or to temporarily switch the green light to a flashing yellow light to give a high-intensity warning to the vehicle to prompt the vehicle to decelerate and avoid.

[0082] As Figure 5 shown, a pedestrian detection system based on computer vision provided by an embodiment of the present invention includes:

[0083] A basic model construction module 100, configured to construct an intersection basic model, where the intersection basic model includes at least a road model, a building model, and a device model, and the device model includes at least a model of a binocular camera.

[0084] In this system, the basic model construction module 100 constructs the intersection basic model, performs 3D modeling based on the on-site building structure. When modeling, it at least includes modeling of roads, buildings, and equipment. For example, for intersections with underpass roads, there are often fences on the road surface, and the U-shaped area formed by the fences will block the line of sight of vehicles. When pedestrians pass by, the situation of "ghost poking out" is likely to occur. By setting binocular cameras at the intersection and using the binocular cameras to monitor pedestrians and vehicles to ensure traffic safety. During the modeling process, not only the position of the road and the lane distribution of the road need to be recorded in the model, but also the building model needs to be recorded, including the U-shaped fence. Secondly, the intersection basic model adopts high-precision modeling with a modeling ratio of 1:1. The spatial position and shooting direction of the binocular cameras in the intersection basic model are determined according to the positions of the binocular cameras.

[0085] The standard model generation module 200 is used to collect images through binocular cameras, measure the distances of the collected binocular ranging images, determine the positions of various objects in the picture, and generate the intersection standard model.

[0086] In this system, the standard model generation module 200 collects images through binocular cameras. After the intersection is put into use, there may be many obstacles at the intersection, such as temporarily added guardrails and parked faulty vehicles. Multiple images are collected through binocular cameras, and the objects included in the images are identified through image recognition technology. Secondly, the binocular ranging technology is used to measure the distances between various objects and the binocular cameras. Through multiple image recognitions, if an object is in the same position during multiple image recognition processes, it is recorded in the intersection standard model. Moreover, the size information of various objects, such as height and width, can be calculated using binocular cameras, so as to record the above objects in the intersection standard model and obtain a complete intersection standard model. The intersection standard model is updated every once in a while, such as once a minute.

[0087] The pedestrian trajectory prediction module 300 is used to demarcate the pedestrian detection area in the intersection standard model, identify the pedestrians located in the pedestrian detection area through binocular cameras, mark each pedestrian, and calculate the motion prediction trajectories of each pedestrian.

[0088] In this system, the pedestrian trajectory prediction module 300 demarcates a pedestrian detection area in the intersection standard model. There are zebra crossings at intersections. Based on the set position of the zebra crossing, the pedestrian detection area for pedestrian passage in the intersection standard model is determined. To improve the detection accuracy, the range of the pedestrian detection area can be appropriately expanded, that is, the range of the pedestrian detection area includes at least the range where the crosswalk is located. The binocular camera is used to detect pedestrians within the pedestrian detection area. During this process, face recognition technology is used to identify pedestrians and mark them until the pedestrian disappears from the screen. The pedestrians are identified at preset time intervals to determine the movement trajectory of the pedestrians in the intersection standard model, and a movement prediction trajectory is generated based on the historical movement trajectory. The movement prediction trajectory is the predicted movement trajectory of the pedestrians in the future time period.

[0089] The safety verification module 400 is used to identify each vehicle through the binocular camera, calculate the movement prediction trajectory of the vehicle, conduct line-of-sight occlusion verification and trajectory verification based on the intersection standard model, and issue a warning message based on the verification result.

[0090] In this system, the safety verification module 400 identifies each vehicle through the binocular camera. Similarly, the images collected by the binocular camera also contain vehicle information. The vehicle at the front of each lane is identified, and the binocular ranging technology is used to determine the accurate position of the vehicle in the intersection standard model. The movement prediction trajectory of the vehicle is calculated based on the historical trajectory of the vehicle. Then, based on the movement prediction trajectory of the vehicle and the movement prediction trajectory of the pedestrian, the trajectory verification can be completed to determine whether there will be a coincidence point between the two sets of trajectories. And during this process, the line-of-sight occlusion verification is carried out. Since there are obstacles in the intersection, such as U-shaped enclosures, pedestrians and vehicles appear on the two right-angled sides of the U-shaped enclosure respectively. The right-angled side of the U-shaped enclosure or other obstacles may block the pedestrians. Therefore, according to the positions of the vehicle, the pedestrian, and the obstacle, it is determined whether the pedestrian will be blocked by the obstacle to generate a verification result. When the verification result shows that the pedestrian will be blocked, a warning message is generated to warn the vehicle and the pedestrian through sound and light warnings to improve road traffic safety.

[0091] As Figure 6 shown, as a preferred embodiment of the present invention, the standard model generation module 200 includes:

[0092] The object recognition unit 201 is used to collect an image to be measured through the binocular camera and identify each object in the image to be measured.

[0093] In this module, the object recognition unit 201 acquires the image to be measured through a binocular camera. When performing detection, the image to be measured is acquired at a preset time interval. Specifically, multiple groups of images to be measured can be acquired within one minute, such as 10 groups of images to be measured. The objects included in the image are recognized through an image recognition algorithm, and the types of the recognized objects are determined.

[0094] The distance value calculation unit 202 is used to calculate the reference distance value between each object and the fixed reference object in the intersection by using the binocular ranging algorithm.

[0095] The standard model generation unit 203 is used to retrieve the basic intersection model and generate the dynamic models of each object in the basic intersection model based on the reference distance value to obtain the standard intersection model.

[0096] In this module, the distance between the object and the binocular camera is calculated by using the binocular ranging algorithm. By using the binocular ranging technology, the size information of each object in the picture can be determined, and in the standard intersection model, the specific positions of each object model are recorded, such as the specific position of the U-shaped enclosure. Therefore, the positions of each object, vehicle, and pedestrian in the standard intersection model can be determined.

[0097] Such as Figure 7 As shown, as a preferred embodiment of the present invention, the pedestrian trajectory prediction module 300 includes:

[0098] The area division unit 301 is used to set a pedestrian detection area in the standard intersection model based on the passing direction of the pedestrian, and the pedestrian detection area covers the crosswalk.

[0099] In this module, the area division unit 301 sets a pedestrian detection area in the standard intersection model based on the passing direction of the pedestrian. In the road, a crosswalk is set in any passing direction, and accordingly, a pedestrian detection area is set in the standard intersection model.

[0100] The image acquisition unit 302 is used to acquire images at a preset acquisition time interval to obtain the binocular images to be detected, and the binocular images to be detected include two groups of actual captured images, and the two groups of actual captured images come from two cameras in the binocular camera respectively.

[0101] In this module, the image acquisition unit 302 acquires images at a preset acquisition time interval, such as acquiring images once every 50 ms, to obtain a group of binocular images to be detected. The binocular camera actually includes two cameras. Therefore, when acquiring images, two groups of actual captured images will be obtained in one shot, that is, one camera generates one group of actual captured images.

[0102] The pedestrian recognition unit 303 is used to recognize pedestrians in the picture by using the face recognition algorithm, mark each pedestrian, determine the movement trajectory of the pedestrian according to the consecutive binocular images to be detected, generate a motion prediction trajectory, and generate pedestrian information.

[0103] In this module, the pedestrian recognition unit 303 uses the face recognition algorithm to recognize pedestrians in the picture and determine the positions of the pedestrians in the picture. In order to distinguish the pedestrians, a mark is set for each pedestrian. Specifically, it can be a code, such as pedestrian No. A00001. When a pedestrian appears in the picture, the pedestrian is always marked with this code. When the pedestrian no longer appears in the picture, a new code is assigned to it when it enters again. The binocular images to be detected are collected at time intervals. Therefore, each group of binocular images to be detected will determine a position of each pedestrian in the intersection standard model once. Then, in the intersection standard model, the historical movement trajectory of the pedestrian can be determined. Based on the movement trend of the historical movement trajectory, the motion prediction trajectory of the pedestrian in the short term in the future can be determined. Specifically, a two-dimensional coordinate system can be constructed based on the ground of the intersection standard model, so as to mark the historical positions of the pedestrians in the two-dimensional coordinate system. Function fitting is performed according to the coordinate points corresponding to the above historical positions (the abscissa is the time value, and the ordinate is the distance value of the pedestrian from one end of the zebra crossing) to obtain the pedestrian movement trajectory function. Based on the above pedestrian movement trajectory function, the motion prediction trajectory of the pedestrian in the short term can be determined, and pedestrian information is generated. The pedestrian information at least includes the height of the pedestrian and the position information of the pedestrian in the horizontal plane.

[0104] As Figure 8 shown, as a preferred embodiment of the present invention, the safety verification module 400 includes:

[0105] The vehicle recognition unit 401 is used to retrieve the binocular images to be detected collected by the binocular camera, recognize the vehicles contained therein, generate vehicle information, and the vehicle information at least includes vehicle type, vehicle size, and driver's seat height.

[0106] In this module, the vehicle recognition unit 401 retrieves the binocular images to be detected captured by the binocular camera. In the binocular images to be detected, not only pedestrians are included, but also vehicles are included in each lane. Similarly, using image recognition technology, the license plate numbers, vehicle sizes, and vehicle types of each vehicle in the image are recognized. The binocular camera can calculate the depth information of the objects in the image, that is, the distance from the camera, by simulating the principle of stereoscopic vision of the human eye and capturing images of the same scene from different positions with two cameras. After obtaining the depth information of the object and the vertical height from the camera to the ground, combined with geometric relationships, the size information such as the height or width of the object or task in the image can be calculated. Taking the calculation of the human height as an example, the depth information is obtained as follows: First, by matching the left and right images captured by the binocular camera and using the triangulation method to calculate the depth of the object corresponding to each pixel point in the image (i.e., the Z-axis distance); determining the camera height: obtaining the accurate height of the binocular camera relative to the ground; measuring the pixel coordinates of the head and feet of the person: identifying and marking the positions of the head and feet of the person in the image and obtaining the pixel coordinates of these two points in the image; calculating the actual height: using the principle of similar triangles, the human height (H) can be estimated by the following formula:

[0107]

[0108] where h 0 is the height of the binocular camera minus the vertical distance from the bottom of the camera to the ground, h p is the pixel difference from the feet to the head of the person in the image, and l s is the pixel distance corresponding to the depth value from the camera to the plane where the person stands. Based on this, the height of the person can be calculated. Based on the same principle, the width of the person, the height and width of the vehicle can be calculated. Based on the position of the face in the image, the position of the driver's seat can be determined, so as to obtain vehicle information and pedestrian information.

[0109] The trajectory verification unit 402 is used to determine the positions of the pedestrians and vehicles in the intersection standard model based on the vehicle information and pedestrian information, and determine the movement trajectories of the pedestrians and vehicles, complete the line-of-sight occlusion verification and trajectory verification, and obtain the verification result.

[0110] The risk warning unit 403 is used to issue a warning message when it is determined based on the verification result that there is a line-of-sight occlusion between the vehicle and the pedestrian and their trajectories overlap.

[0111] In this module, the trajectory verification unit 402 determines the positions of the pedestrian and the vehicle in the intersection standard model based on vehicle information and pedestrian information. Specifically, since the intersection standard model is a three-dimensional model, a three-dimensional coordinate system can be constructed based on this intersection standard model. And the predicted movement trajectories of the pedestrian and the vehicle have been determined. Then, first, a trajectory overlap judgment is performed. The two trajectories are plotted in the three-dimensional coordinate system to determine the positions of the pedestrian and the vehicle at each moment. If, at a certain moment, the distance between the positions of the pedestrian and the vehicle is less than a preset value, it is determined that there is a trajectory overlap. Subsequently, a line-of-sight occlusion verification is started. The position of the driver's seat is determined based on vehicle information, and the coordinates of the driver's seat are determined. The steps include: Image acquisition: The binocular camera simultaneously captures the same scene from slightly different perspectives; Disparity calculation: By comparing the pixel position differences (disparities) of the same object in the images captured by the two cameras, the distance of the object from the camera can be known. The larger the disparity, the closer the object is to the camera; the smaller the disparity, the farther the object is; Three-dimensional reconstruction: Using the disparity information and the known baseline distance between the cameras (the distance between the centers of the two cameras), combined with the internal parameters of the camera (such as the focal length) and external parameters (such as the position and orientation of the camera), the coordinates of the object corresponding to each pixel in the image in the three-dimensional space are calculated through triangulation; Depth information and position calculation: Once the depth information of each pixel is obtained, the three-dimensional structure of the entire scene, including the spatial positions (X, Y, Z coordinates) of the objects, can be reconstructed. In the same way, the coordinates of the pedestrian are determined. To ensure reliability, the coordinates at half of the pedestrian's height are used as the coordinates of the pedestrian. A line is connected between the pedestrian coordinates and the driver's seat coordinates. If the connection between the two is blocked by an obstacle, it is determined that there is a line-of-sight obstacle; otherwise, it is determined that there is no line-of-sight obstacle. When it is determined that there are both line-of-sight obstacles and trajectory overlaps, an alarm is issued. The alarm method can be to warn the vehicle and the pedestrian through sound and light signals, or to temporarily switch the green light to a flashing yellow light to highly warn the vehicle to prompt the vehicle to decelerate and avoid.

[0112] It should be understood that although the steps in the flowcharts of the embodiments of the present invention are shown sequentially according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same moment, but can be executed at different moments. The execution order of these sub-steps or stages does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0113] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0114] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0115] The above embodiments merely represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the invention patent of the present invention shall be subject to the appended claims.

[0116] The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A pedestrian detection method based on computer vision, characterized in that: The method comprises: Constructing a basic intersection model, wherein the basic intersection model includes a road model, a building model, and a device model, wherein the device model includes a model of a binocular camera; The binocular camera is used to collect images, and the distance is calculated for the collected binocular ranging images to determine the location of each object in the picture and generate a standard intersection model; A pedestrian detection area is defined in the standard intersection model, and pedestrians in the pedestrian detection area are identified by a binocular camera, each pedestrian is marked, and the predicted motion trajectory of each pedestrian is calculated; Identify each vehicle through binocular cameras, calculate the predicted movement trajectory of the vehicle, perform line of sight occlusion verification and trajectory verification based on the standard intersection model, and issue warning information based on the verification results; The steps of defining a pedestrian detection area in the standard intersection model, identifying pedestrians in the pedestrian detection area by using a binocular camera, marking each pedestrian, and calculating the predicted motion trajectory of each pedestrian specifically include: Based on the pedestrian's travel direction, a pedestrian detection area is set in the intersection standard model, and the pedestrian detection area covers the crosswalk; Perform image acquisition at a preset acquisition time interval to obtain a binocular image to be detected, wherein the binocular image to be detected includes two groups of real-shot images, and the two groups of real-shot images are respectively from two cameras in the binocular camera; Use the face recognition algorithm to identify pedestrians in the picture, mark each pedestrian, determine the pedestrian's motion trajectory based on the continuous binocular images to be detected, generate a motion prediction trajectory, and generate pedestrian information; The steps of identifying each vehicle by a binocular camera, calculating the predicted motion trajectory of the vehicle, performing sight blockage verification and trajectory verification based on the intersection standard model, and issuing a warning message based on the verification result specifically include: Retrieving a binocular image to be detected acquired by a binocular camera, identifying the vehicle contained therein, and generating vehicle information, wherein the vehicle information includes vehicle type, vehicle size, and driving position height; Based on vehicle information and pedestrian information, the positions of pedestrians and vehicles in the standard model of the intersection are determined, and the movement trajectories of pedestrians and vehicles are determined, and the line of sight occlusion verification and trajectory verification are completed to obtain the verification results; If it is determined based on the verification results that there is a visual obstruction between the vehicle and the pedestrian and their trajectories overlap, a warning message is issued; The positions of pedestrians and vehicles in the standard model of the intersection are determined based on vehicle information and pedestrian information. Specifically, the standard model of the intersection is a three-dimensional model. A three-dimensional coordinate system is constructed based on the standard model of the intersection to determine the predicted motion trajectory of the pedestrian and the predicted motion trajectory of the vehicle. A trajectory overlap judgment is performed, and two trajectories are drawn in the three-dimensional coordinate system to determine the positions of pedestrians and vehicles at each moment. If at a certain moment, the distance between the positions of the pedestrians and the vehicles is less than a preset value, it is determined that there is a trajectory overlap, and then a line of sight occlusion check is performed to determine the position of the driving seat based on the vehicle information, determine the coordinates of the driving seat and the coordinates of the pedestrian, and use the coordinates at half the height of the pedestrian as the coordinates of the pedestrian. A line is drawn based on the pedestrian coordinates and the driver's seat coordinates. If the line connecting the two is blocked by an obstacle, it is determined that there is a line of sight obstruction. Otherwise, it is determined that there is no line of sight obstruction. An alarm is issued when it is determined that there is both a line of sight obstruction and a trajectory overlap.

2. The method for pedestrian detection based on computer vision according to claim 1, characterized in that: The steps of collecting images through a binocular camera, measuring distances on the collected binocular ranging images, determining the positions of various objects in the image, and generating a standard intersection model specifically include: The image to be measured is acquired through the binocular camera, and each object in the image to be measured is identified; The reference distance value between each object and the fixed reference object in the intersection is calculated using the binocular ranging algorithm; The basic intersection model is retrieved, and the dynamic models of various objects are generated in the basic intersection model based on the reference distance value to obtain the standard intersection model.

3. The method for pedestrian detection based on computer vision according to claim 1, characterized in that: The step of issuing a warning message includes sending a control signal to a traffic light to warn the vehicle via the traffic light.

4. A pedestrian detection system based on computer vision, characterized in that: The system comprises: A basic model building module, used to build a basic intersection model, wherein the basic intersection model includes a road model, a building model, and an equipment model, wherein the equipment model includes a model of a binocular camera; The standard model generation module is used to collect images through a binocular camera, measure the distance of the collected binocular ranging images, determine the position of each object in the picture, and generate a standard model of the intersection; The pedestrian trajectory prediction module is used to define the pedestrian detection area in the standard intersection model, identify pedestrians in the pedestrian detection area through a binocular camera, mark each pedestrian, and calculate the predicted motion trajectory of each pedestrian; The safety verification module is used to identify each vehicle through a binocular camera, calculate the vehicle's predicted motion trajectory, perform line of sight occlusion verification and trajectory verification based on the intersection standard model, and issue warning information based on the verification results; The pedestrian trajectory prediction module includes: A region division unit, configured to set a pedestrian detection region in a standard intersection model based on a pedestrian's travel direction, wherein the pedestrian detection region covers a crosswalk; An image acquisition unit is used to acquire images at a preset acquisition time interval to obtain a binocular image to be detected, wherein the binocular image to be detected includes two groups of real-shot images, and the two groups of real-shot images are respectively from two cameras in the binocular camera; A pedestrian recognition unit is used to identify pedestrians in the picture using a face recognition algorithm, mark each pedestrian, determine the movement trajectory of the pedestrian based on the continuous binocular images to be detected, generate a movement prediction trajectory, and generate pedestrian information; The safety verification module comprises: A vehicle identification unit is used to retrieve the binocular image to be detected collected by the binocular camera, identify the vehicle contained therein, and generate vehicle information, wherein the vehicle information includes vehicle type, vehicle size, and driving position height; A trajectory verification unit is used to determine the positions of pedestrians and vehicles in the standard model of the intersection based on vehicle information and pedestrian information, and to determine the movement trajectories of pedestrians and vehicles, complete line of sight occlusion verification and trajectory verification, and obtain verification results; A risk warning unit is used to issue a warning message when it is determined based on the verification result that there is a line of sight obstruction between the vehicle and the pedestrian and their trajectories overlap; The positions of pedestrians and vehicles in the standard model of the intersection are determined based on vehicle information and pedestrian information. Specifically, the standard model of the intersection is a three-dimensional model. A three-dimensional coordinate system is constructed based on the standard model of the intersection to determine the predicted motion trajectory of the pedestrian and the predicted motion trajectory of the vehicle. A trajectory overlap judgment is performed, and two trajectories are drawn in the three-dimensional coordinate system to determine the positions of pedestrians and vehicles at each moment. If at a certain moment, the distance between the positions of the pedestrians and the vehicles is less than a preset value, it is determined that there is a trajectory overlap, and then a line of sight occlusion check is performed to determine the position of the driving seat based on the vehicle information, determine the coordinates of the driving seat and the coordinates of the pedestrian, and use the coordinates at half the height of the pedestrian as the coordinates of the pedestrian. A line is drawn based on the pedestrian coordinates and the driver's seat coordinates. If the line connecting the two is blocked by an obstacle, it is determined that there is a line of sight obstruction. Otherwise, it is determined that there is no line of sight obstruction. An alarm is issued when it is determined that there is both a line of sight obstruction and a trajectory overlap.

5. The computer vision-based pedestrian detection system according to claim 4, characterized in that: The standard model generation module includes: An object recognition unit is used to acquire images to be measured and calculated by using a binocular camera, and to recognize each object in the image to be measured and calculated; A distance value calculation unit, used to calculate the reference distance value between each object and a fixed reference object in the intersection by using a binocular distance measurement algorithm; The standard model generation unit is used to retrieve the basic intersection model, generate dynamic models of various objects in the basic intersection model based on the reference distance value, and obtain the standard intersection model.

6. The computer vision-based pedestrian detection system according to claim 4, characterized in that: The step of issuing a warning message includes sending a control signal to a traffic light to warn the vehicle via the traffic light.

Images