Pedestrian detection method and system based on computer vision

Abstract

The invention is suitable for the technical field of pedestrian detection, and particularly relates to a pedestrian detection method and system based on computer vision, and the method comprises the steps: constructing an intersection basic model; carrying out image acquisition through a binocular camera, carrying out distance measurement and calculation, determining the position of each object in a picture, and generating an intersection standard model; defining a pedestrian detection area, identifying pedestrians in the pedestrian detection area through a binocular camera, marking each pedestrian, and calculating a motion prediction track of each pedestrian; and identifying each vehicle, calculating a motion prediction track of the vehicle, carrying out sight shielding checking and track checking, and sending out warning information based on a checking result. According to the invention, the binocular camera is used for determining the driving tracks of pedestrians and vehicles, and when the sight is blocked, the vehicle is warned through the warning information, so that traffic accidents caused by the sight blocking are avoided, and the traffic safety is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian detection, and in particular relates to a pedestrian detection method and system based on computer vision. Background Art

[0002] Pedestrian detection, as a key component of computer vision technology, aims to automatically identify and accurately locate pedestrians from images or videos.

[0003] In the process of traffic management, warning devices are often installed at intersections to warn passing pedestrians when the red light is on. The existing warning devices can warn pedestrians, but cannot remind drivers when there are visual impairments. Therefore, when pedestrians fail to cross the crosswalk within the time period when the green light is on, the starting and passage of vehicles will pose a threat to pedestrians. Summary of the invention

[0004] The purpose of the present invention is to provide a pedestrian detection method based on computer vision, which aims to solve the problem that when pedestrians fail to cross the crosswalk within the time period when the green light is on, the starting and passing of vehicles will pose a threat to pedestrians.

[0005] The present invention is implemented in a method for detecting pedestrians based on computer vision, the method comprising:

[0006] Constructing a basic intersection model, wherein the basic intersection model at least includes a road model, a building model, and a device model, wherein the device model at least includes a model of a binocular camera;

[0007] The binocular camera is used to collect images, and the distance is calculated for the collected binocular ranging images to determine the location of each object in the picture and generate a standard intersection model;

[0008] A pedestrian detection area is defined in the standard intersection model, and pedestrians in the pedestrian detection area are identified by a binocular camera, each pedestrian is marked, and the predicted motion trajectory of each pedestrian is calculated;

[0009] Each vehicle is identified through a binocular camera, the predicted motion trajectory of the vehicle is calculated, line of sight obstruction verification and trajectory verification are performed based on the standard intersection model, and warning information is issued based on the verification results.

[0010] Preferably, the step of collecting images by a binocular camera, measuring distances on the collected binocular ranging images, determining the positions of various objects in the picture, and generating a standard model of the intersection specifically includes:

[0011] The image to be measured is acquired through the binocular camera, and each object in the image to be measured is identified;

[0012] The reference distance value between each object and the fixed reference object in the intersection is calculated using the binocular ranging algorithm;

[0013] The basic intersection model is retrieved, and the dynamic models of various objects are generated in the basic intersection model based on the reference distance value to obtain the standard intersection model.

[0014] Preferably, the steps of defining a pedestrian detection area in the standard intersection model, identifying pedestrians in the pedestrian detection area by using a binocular camera, marking each pedestrian, and calculating the predicted motion trajectory of each pedestrian specifically include:

[0015] Based on the pedestrian's travel direction, a pedestrian detection area is set in the intersection standard model, and the pedestrian detection area covers the crosswalk;

[0016] Perform image acquisition at a preset acquisition time interval to obtain a binocular image to be detected, wherein the binocular image to be detected includes two groups of real-shot images, and the two groups of real-shot images are respectively from two cameras in the binocular camera;

[0017] The face recognition algorithm is used to identify pedestrians in the picture, mark each pedestrian, determine the pedestrian's motion trajectory based on the continuous binocular images to be detected, generate a motion prediction trajectory, and generate pedestrian information.

[0018] Preferably, the steps of identifying each vehicle by a binocular camera, calculating the predicted motion trajectory of the vehicle, performing sight blockage verification and trajectory verification based on a standard intersection model, and issuing a warning message based on the verification result specifically include:

[0019] Retrieving a binocular image to be detected acquired by a binocular camera, identifying the vehicle contained therein, and generating vehicle information, wherein the vehicle information includes at least the vehicle type, vehicle size, and driving position height;

[0020] Based on vehicle information and pedestrian information, the positions of pedestrians and vehicles in the standard model of the intersection are determined, and the movement trajectories of pedestrians and vehicles are determined, and the line of sight occlusion verification and trajectory verification are completed to obtain the verification results;

[0021] When it is determined based on the verification results that there is a visual obstruction between the vehicle and the pedestrian and their trajectories overlap, a warning message is issued.

[0022] Preferably, the step of issuing a warning message includes sending a control signal to a traffic light to warn the vehicle via the traffic light.

[0023] Another object of the present invention is to provide a pedestrian detection system based on computer vision, the system comprising:

[0024] A basic model building module, used to build a basic intersection model, wherein the basic intersection model includes at least a road model, a building model and a device model, wherein the device model includes at least a model of a binocular camera;

[0025] The standard model generation module is used to collect images through a binocular camera, measure the distance of the collected binocular ranging images, determine the position of each object in the picture, and generate a standard model of the intersection;

[0026] The pedestrian trajectory prediction module is used to define the pedestrian detection area in the standard intersection model, identify pedestrians in the pedestrian detection area through a binocular camera, mark each pedestrian, and calculate the predicted motion trajectory of each pedestrian;

[0027] The safety verification module is used to identify each vehicle through a binocular camera, calculate the vehicle's predicted motion trajectory, perform line of sight occlusion verification and trajectory verification based on the standard intersection model, and issue warning information based on the verification results.

[0028] Preferably, the standard model generation module includes:

[0029] An object recognition unit is used to acquire images to be measured and calculated by using a binocular camera, and to recognize each object in the image to be measured and calculated;

[0030] A distance value calculation unit, used to calculate the reference distance value between each object and a fixed reference object in the intersection by using a binocular distance measurement algorithm;

[0031] The standard model generation unit is used to retrieve the basic intersection model, generate dynamic models of various objects in the basic intersection model based on the reference distance value, and obtain the standard intersection model.

[0032] Preferably, the pedestrian trajectory prediction module includes:

[0033] A region division unit, configured to set a pedestrian detection region in a standard intersection model based on a pedestrian's travel direction, wherein the pedestrian detection region covers a crosswalk;

[0034] An image acquisition unit is used to acquire images at a preset acquisition time interval to obtain a binocular image to be detected, wherein the binocular image to be detected includes two groups of real-shot images, and the two groups of real-shot images are respectively from two cameras in the binocular camera;

[0035] The pedestrian recognition unit is used to use the face recognition algorithm to identify pedestrians in the picture, mark each pedestrian, determine the movement trajectory of the pedestrian based on the continuous binocular images to be detected, generate the movement prediction trajectory, and generate pedestrian information.

[0036] Preferably, the safety verification module includes:

[0037] A vehicle identification unit is used to retrieve the binocular image to be detected collected by the binocular camera, identify the vehicle contained therein, and generate vehicle information, wherein the vehicle information at least includes the vehicle type, vehicle size, and driving position height;

[0038] A trajectory verification unit is used to determine the positions of pedestrians and vehicles in the standard model of the intersection based on vehicle information and pedestrian information, and to determine the movement trajectories of pedestrians and vehicles, complete line of sight occlusion verification and trajectory verification, and obtain verification results;

[0039] The risk warning unit is used to issue a warning message when it is determined based on the verification result that there is a visual obstruction between the vehicle and the pedestrian and their trajectories overlap.

[0040] Preferably, the step of issuing a warning message includes sending a control signal to a traffic light to warn the vehicle via the traffic light.

[0041] The present invention provides a pedestrian detection method based on computer vision. By constructing a standard intersection model, the distribution of basic building structures at the intersection can be determined, and then the driving trajectories of pedestrians and vehicles can be determined using a binocular camera. When there is line of sight obstruction, the vehicle is warned through warning information, thereby avoiding traffic accidents caused by line of sight obstruction and improving traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a pedestrian detection method based on computer vision provided by an embodiment of the present invention;

[0043] Figure 2 A flowchart of the steps of acquiring images through a binocular camera, measuring distances on acquired binocular ranging images, determining the positions of various objects in the image, and generating a standard model of an intersection provided by an embodiment of the present invention;

[0044] Figure 3 A flowchart of the steps of defining a pedestrian detection area in a standard intersection model, identifying pedestrians in the pedestrian detection area by a binocular camera, marking each pedestrian, and calculating a predicted motion trajectory of each pedestrian provided by an embodiment of the present invention;

[0045] Figure 4 A flowchart of the steps of identifying each vehicle by a binocular camera, calculating the predicted motion trajectory of the vehicle, performing line of sight occlusion verification and trajectory verification based on a standard intersection model, and issuing a warning message based on the verification result provided by an embodiment of the present invention;

[0046] Figure 5 An architecture diagram of a pedestrian detection system based on computer vision provided by an embodiment of the present invention;

[0047] Figure 6 An architectural diagram of a standard model generation module provided by an embodiment of the present invention;

[0048] Figure 7 An architecture diagram of a pedestrian trajectory prediction module provided by an embodiment of the present invention;

[0049] Figure 8 An architectural diagram of a security verification module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0052] like Figure 1 FIG. 1 is a flowchart of a method for detecting pedestrians based on computer vision provided by an embodiment of the present invention, wherein the method comprises:

[0053] S100, constructing a basic intersection model, wherein the basic intersection model at least includes a road model, a building model, and a device model, and the device model at least includes a model of a binocular camera.

[0054] In this step, a basic model of the intersection is constructed, and three-dimensional modeling is performed based on the on-site building structure. When performing modeling, at least the modeling of the road, the modeling of the building, and the modeling of the equipment are included. For example, for intersections with underpasses, the road surface is often equipped with fences, and the U-shaped area formed by the fences will block the line of sight of the vehicle. When pedestrians pass by, it is easy for ghosts to appear. By setting up binocular cameras at the intersection, pedestrians and vehicles are monitored by binocular cameras to ensure traffic safety. In the modeling process, not only the location of the road and the lane distribution of the road should be recorded in the model, but also the building model, including the U-shaped fence. Secondly, the basic model of the intersection adopts high-precision modeling with a modeling ratio of 1:1. The spatial position of the binocular camera in the basic model of the intersection and the shooting direction are determined according to the position of the binocular camera.

[0055] S200, collecting images through a binocular camera, measuring distances on the collected binocular ranging images, determining positions of various objects in the image, and generating a standard intersection model.

[0056] In this step, image acquisition is performed through a binocular camera. After the intersection is put into use, there may be more obstacles at the intersection, such as temporarily added guardrails and parked faulty vehicles. Multiple image acquisitions are performed through the binocular camera, and the objects contained in the image are identified through image recognition technology. Secondly, the distance between each object and the binocular camera is measured using binocular ranging technology. Through multiple image recognitions, if the object is in the same position during multiple image recognition processes, it will be recorded in the standard intersection model. In addition, the binocular camera can be used to calculate the size information of each object, such as height and width, so as to record the above objects in the standard intersection model to obtain a complete standard intersection model. The standard intersection model is updated once every period of time, such as once a minute.

[0057] S300, delineating a pedestrian detection area in the standard intersection model, identifying pedestrians in the pedestrian detection area through a binocular camera, marking each pedestrian, and calculating a predicted motion trajectory of each pedestrian.

[0058] In this step, a pedestrian detection area is delineated in the standard model of the intersection. Zebra crossings are set at intersections. The pedestrian detection area for pedestrian passage in the standard model of the intersection is determined based on the position of the zebra crossing. In order to improve the detection accuracy, the range of the pedestrian detection area can be appropriately expanded, that is, the range of the pedestrian detection area at least includes the range of the crosswalk. Pedestrians in the pedestrian detection area are detected using a binocular camera. In this process, face recognition technology is used to identify pedestrians and mark them until they disappear from the picture. Pedestrians are identified at preset time intervals to determine the movement trajectory of pedestrians in the standard model of the intersection, and a motion prediction trajectory is generated based on the historical motion trajectory. The motion prediction trajectory is the predicted movement trajectory of the pedestrian in the future time period.

[0059] S400 uses binocular cameras to identify each vehicle, calculates the vehicle's predicted motion trajectory, performs line of sight occlusion verification and trajectory verification based on the intersection standard model, and issues warning information based on the verification results.

[0060] In this step, each vehicle is identified by a binocular camera. Similarly, the image collected by the binocular camera also contains vehicle information. The vehicle at the front of each lane is identified, and the binocular ranging technology is used to determine the exact position of the vehicle in the standard model of the intersection. The predicted motion trajectory of the vehicle is calculated based on the historical trajectory of the vehicle. Then, based on the predicted motion trajectory of the vehicle and the predicted motion trajectory of the pedestrian, the trajectory verification can be completed to determine whether there will be overlapping points in the two sets of trajectories, and a line of sight occlusion verification is performed in the process. Since there are obstacles at the intersection, such as U-shaped fences, pedestrians and vehicles appear at the two right-angled sides of the U-shaped fence respectively. The right-angled sides of the U-shaped fence or other obstacles may block pedestrians. Therefore, it is determined whether the pedestrian will be blocked by the obstacle based on the position of the vehicle, the position of the pedestrian and the position of the obstacle, and a verification result is generated. When the verification result shows that the pedestrian will be blocked, a warning message is generated, and the vehicle and pedestrian are warned through sound and light warnings to improve road traffic safety.

[0061] like Figure 2 As shown, as a preferred embodiment of the present invention, the steps of collecting images by binocular cameras, measuring distances on the collected binocular ranging images, determining the positions of various objects in the picture, and generating a standard model of the intersection specifically include:

[0062] S201, acquiring an image to be measured by using a binocular camera, and identifying each object in the image to be measured.

[0063] In this step, the image to be measured is obtained by image acquisition through a binocular camera. When performing detection, the image to be measured is acquired at preset time intervals. Specifically, multiple groups of images to be measured can be acquired within one minute, such as 10 groups of images to be measured. The objects contained in the images are identified through an image recognition algorithm to identify the type of the object.

[0064] S202: Calculate the reference distance value between each object and the fixed reference object in the intersection by using a binocular distance measurement algorithm.

[0065] S203, retrieve the basic intersection model, generate dynamic models of various objects in the basic intersection model based on the reference distance value, and obtain a standard intersection model.

[0066] In this step, the binocular ranging algorithm is used to calculate the distance between the object and the binocular camera. The binocular ranging technology can be used to determine the size information of each object in the picture, and the specific position of each object model is recorded in the standard intersection model, such as the specific position of the U-shaped fence. Therefore, the position of each object, vehicle and pedestrian in the standard intersection model can be determined.

[0067] like Figure 3As shown, as a preferred embodiment of the present invention, the steps of defining a pedestrian detection area in the standard intersection model, identifying pedestrians in the pedestrian detection area by a binocular camera, marking each pedestrian, and calculating the predicted motion trajectory of each pedestrian specifically include:

[0068] S301, based on the pedestrian's travel direction, a pedestrian detection area is set in the intersection standard model, and the pedestrian detection area covers the crosswalk.

[0069] In this step, a pedestrian detection area is set in the standard intersection model based on the pedestrian's travel direction. On the road, a crosswalk is set in any travel direction, and a pedestrian detection area is set in the standard intersection model accordingly.

[0070] S302, performing image acquisition at a preset acquisition time interval to obtain a binocular image to be detected, wherein the binocular image to be detected includes two groups of real-shot images, and the two groups of real-shot images are respectively from two cameras in the binocular camera.

[0071] In this step, image acquisition is performed according to a preset acquisition time interval, such as once every 50ms, to obtain a set of binocular images to be detected. The binocular camera actually contains two cameras, so when acquiring images, two sets of real-shot images will be obtained at one time, that is, one camera generates a set of real-shot images.

[0072] S303, using a face recognition algorithm to identify pedestrians in the picture, marking each pedestrian, determining the movement trajectory of the pedestrian according to the continuous binocular images to be detected, generating a movement prediction trajectory, and generating pedestrian information.

[0073] In this step, the face recognition algorithm is used to identify the pedestrians in the picture and determine the position of the pedestrians in the picture. In order to distinguish the pedestrians, a mark is set for each pedestrian. Specifically, it can be a code, such as pedestrian A00001. When the pedestrian appears in the picture, it is always marked with the code. When the pedestrian no longer appears in the picture, it is given a new code when it enters again. The binocular images to be detected are collected at time intervals. Therefore, each group of binocular images to be detected will determine a position of each pedestrian in the standard model of the intersection once, so the history of the pedestrian can be determined in the standard model of the intersection. Motion trajectory, based on the motion trend of the historical motion trajectory, the predicted motion trajectory of the pedestrian in the short future time can be determined. Specifically, a two-dimensional coordinate system can be constructed based on the ground of the standard model of the intersection, so that the historical position of the pedestrian can be marked in the two-dimensional coordinate system, and the function is fitted according to the coordinate points corresponding to the above historical positions (the horizontal axis is the time value, and the vertical axis is the distance value of the pedestrian from one end of the zebra crossing) to obtain the pedestrian motion trajectory function. Based on the above pedestrian motion trajectory function, the predicted motion trajectory of the pedestrian in a short time can be determined, and pedestrian information can be generated. The pedestrian information at least includes the height of the pedestrian and the position information of the pedestrian in the horizontal plane.

[0074] like Figure 4 As shown, as a preferred embodiment of the present invention, the steps of identifying each vehicle by a binocular camera, calculating the predicted motion trajectory of the vehicle, performing line of sight occlusion verification and trajectory verification based on the intersection standard model, and issuing a warning message based on the verification result specifically include:

[0075] S401, retrieve the binocular image to be detected collected by the binocular camera, identify the vehicle contained therein, and generate vehicle information, wherein the vehicle information at least includes the vehicle type, vehicle size, and driving position height.

[0076] In this step, the binocular image to be detected collected by the binocular camera is retrieved. The binocular image to be detected includes not only pedestrians but also vehicles in each lane. Similarly, the image recognition technology is used to identify the license plate number, vehicle size and vehicle type of each vehicle in the picture. The binocular camera can simulate the stereoscopic vision principle of the human eye and use two cameras to capture images of the same scene from different positions. The depth information of the object in the image, that is, the distance from the camera, can be calculated. After obtaining the depth information of the object and the vertical height from the camera to the ground, combined with the geometric relationship, the depth information of the object in the image can be calculated. The height or width of the object or task is obtained by calculating the height of the person. For example, the depth information is obtained by matching the left and right images captured by the binocular camera and using triangulation to calculate the object depth (i.e., the Z-axis distance) corresponding to each pixel in the image. Determine the camera height: obtain the precise height of the binocular camera relative to the ground. Measure the pixel coordinates of the top of the head and feet of the person: identify and mark the top of the head and feet of the person in the image, and obtain the pixel coordinates of these two points in the image. Calculate the actual height: Using the principle of similar triangles, the height of the person (H) can be estimated by the following formula:

[0077]

[0078] Among them, h 0 h is the height of the binocular camera minus the vertical distance from the bottom of the camera to the ground. p is the pixel difference from the feet to the top of the head in the image, l s It is the pixel distance corresponding to the depth value from the camera to the plane where the person is standing. Based on this, the height of the person can be calculated. Based on the same principle, the width of the person, the height and width of the vehicle can be calculated. Based on the position of the face in the picture, the position of the driving seat can be determined, thereby obtaining vehicle information and pedestrian information.

[0079] S402, based on the vehicle information and pedestrian information, determine the positions of pedestrians and vehicles in the standard model of the intersection, and determine the movement trajectories of pedestrians and vehicles, complete the line of sight occlusion verification and trajectory verification, and obtain the verification results.

[0080] S403: When it is determined based on the verification result that there is a visual obstruction between the vehicle and the pedestrian and their trajectories overlap, a warning message is issued.

[0081] In this step, the positions of pedestrians and vehicles in the standard model of the intersection are determined based on the vehicle information and pedestrian information. Specifically, the standard model of the intersection is a three-dimensional model, so a three-dimensional coordinate system can be constructed based on the standard model of the intersection, and the predicted motion trajectory of the pedestrian and the predicted motion trajectory of the vehicle have been determined. Then, a trajectory overlap judgment is performed first, and two trajectories are drawn in the three-dimensional coordinate system to determine the positions of pedestrians and vehicles at each moment. If at a certain moment, the distance between the positions of pedestrians and vehicles is less than a preset value, it is determined that there is a trajectory overlap, and then a line of sight occlusion check is started. The position of the driving seat is determined based on the vehicle information, and the coordinates of the driving seat are determined. The steps include: image acquisition: binocular cameras simultaneously shoot the same scene from slightly different perspectives; parallax calculation: by comparing the pixel position difference (parallax) of the same object in the images captured by the two cameras, the distance of the object from the camera is known. The larger the parallax, the closer the object is to the camera; the smaller the parallax, the farther the object is; three-dimensional reconstruction: using parallax The information and the known baseline distance between cameras (the distance between the centers of the two cameras) are combined with the internal parameters of the camera (such as focal length) and external parameters (such as the position and orientation of the camera) to calculate the coordinates of the object corresponding to each pixel in the image in three-dimensional space through triangulation. Depth information and position calculation: Once the depth information of each pixel is obtained, the three-dimensional structure of the entire scene can be reconstructed, including the spatial position of the object (X, Y, Z coordinates). In the same way, the coordinates of the pedestrian are determined. In order to ensure reliability, the coordinates at half the height of the pedestrian are used as the coordinates of the pedestrian. A line is drawn based on the coordinates of the pedestrian and the coordinates of the driver's seat. If the line between the two is blocked by an obstacle, it is determined that there is a line of sight obstruction. Otherwise, it is determined that there is no line of sight obstruction. When it is determined that there is both a line of sight obstruction and trajectory overlap, an alarm is issued. The alarm method can be to warn the vehicle and pedestrians through sound and light signals, or to warn the vehicle with a high intensity by temporarily switching the green light to a flashing yellow light to prompt the vehicle to slow down and avoid.

[0082] like Figure 5 As shown, a pedestrian detection system based on computer vision is provided in an embodiment of the present invention, and the system includes:

[0083] The basic model building module 100 is used to build a basic intersection model, wherein the basic intersection model at least includes a road model, a building model and a device model, and the device model at least includes a model of a binocular camera.

[0084] In this system, the basic model construction module 100 constructs a basic model of the intersection and performs three-dimensional modeling based on the on-site building structure. When performing modeling, at least the modeling of the road, the modeling of the building and the modeling of the equipment are included. For example, for an intersection with an underpass, the road surface is often equipped with fences, and the U-shaped area formed by the fences will block the line of sight of the vehicle. When pedestrians pass by, it is easy for ghosts to appear. By setting a binocular camera at the intersection, the binocular camera is used to monitor pedestrians and vehicles to ensure traffic safety. In the modeling process, not only the location of the road and the lane distribution of the road should be recorded in the model, but also the building model, including the U-shaped fence. Secondly, the basic model of the intersection adopts high-precision modeling with a modeling ratio of 1:1. The spatial position of the binocular camera in the basic model of the intersection and the shooting direction are determined according to the position of the binocular camera.

[0085] The standard model generation module 200 is used to collect images through a binocular camera, measure the distance of the collected binocular ranging images, determine the position of each object in the picture, and generate a standard model of the intersection.

[0086] In this system, the standard model generation module 200 uses a binocular camera to collect images. After the intersection is put into use, there may be more obstacles at the intersection, such as temporarily added guardrails and stopped faulty vehicles. The binocular camera is used to collect images multiple times, and the objects contained in the image are identified through image recognition technology. Secondly, the binocular ranging technology is used to measure the distance between each object and the binocular camera. Through multiple image recognitions, if the object is in the same position during multiple image recognition processes, it will be recorded in the intersection standard model. In addition, the binocular camera can be used to calculate the size information of each object, such as height and width, so as to record the above objects in the intersection standard model to obtain a complete intersection standard model. The intersection standard model is updated once every period of time, such as once a minute.

[0087] The pedestrian trajectory prediction module 300 is used to define a pedestrian detection area in the standard intersection model, identify pedestrians in the pedestrian detection area through a binocular camera, mark each pedestrian, and calculate the predicted motion trajectory of each pedestrian.

[0088] In this system, the pedestrian trajectory prediction module 300 delineates a pedestrian detection area in the standard model of the intersection. Zebra crossings are set at intersections. The pedestrian detection area for pedestrian passage in the standard model of the intersection is determined based on the position of the zebra crossing. In order to improve the detection accuracy, the range of the pedestrian detection area can be appropriately expanded, that is, the range of the pedestrian detection area at least includes the range where the crosswalk is located. The binocular camera is used to detect pedestrians in the pedestrian detection area. In this process, face recognition technology is used to identify pedestrians and mark them until they disappear from the picture. Pedestrians are identified at preset time intervals to determine the movement trajectory of pedestrians in the standard model of the intersection, and a motion prediction trajectory is generated based on the historical motion trajectory. The motion prediction trajectory is the predicted movement trajectory of the pedestrian in the future time period.

[0089] The safety verification module 400 is used to identify each vehicle through a binocular camera, calculate the predicted motion trajectory of the vehicle, perform line of sight occlusion verification and trajectory verification based on the intersection standard model, and issue a warning message based on the verification result.

[0090] In this system, the safety verification module 400 identifies each vehicle through a binocular camera. Similarly, the image collected by the binocular camera also contains vehicle information. The vehicle at the front of each lane is identified, and the binocular ranging technology is used to determine the exact position of the vehicle in the standard model of the intersection. The predicted motion trajectory of the vehicle is calculated based on the historical trajectory of the vehicle. Then, based on the predicted motion trajectory of the vehicle and the predicted motion trajectory of the pedestrian, the trajectory verification can be completed to determine whether there will be overlap points in the two sets of trajectories, and a line of sight occlusion verification is performed in the process. Since there are obstacles at the intersection, such as a U-shaped fence, pedestrians and vehicles appear at the two right-angled sides of the U-shaped fence respectively. The right-angled sides of the U-shaped fence or other obstacles may block pedestrians. Therefore, it is determined whether the pedestrian will be blocked by the obstacle based on the position of the vehicle, the position of the pedestrian and the position of the obstacle, and a verification result is generated. When the verification result shows that the pedestrian will be blocked, a warning message is generated, and the vehicle and pedestrian are warned through sound and light warnings to improve road traffic safety.

[0091] like Figure 6 As shown, as a preferred embodiment of the present invention, the standard model generation module 200 includes:

[0092] The object recognition unit 201 is used to acquire an image to be measured by using a binocular camera, and recognize each object in the image to be measured.

[0093] In this module, the object recognition unit 201 acquires images to be measured by using a binocular camera. When performing detection, the images to be measured are acquired at preset time intervals. Specifically, multiple groups of images to be measured can be acquired within one minute, such as 10 groups of images to be measured. The objects contained in the images are identified through an image recognition algorithm to identify the types of the objects.

[0094] The distance value calculation unit 202 is used to calculate the reference distance value between each object and the fixed reference object in the intersection by using a binocular distance measurement algorithm.

[0095] The standard model generation unit 203 is used to retrieve the basic intersection model, generate dynamic models of various objects in the basic intersection model based on the reference distance value, and obtain the standard intersection model.

[0096] In this module, the binocular ranging algorithm is used to calculate the distance between the object and the binocular camera. The binocular ranging technology can determine the size information of each object in the picture, and the specific position of each object model is recorded in the standard intersection model, such as the specific position of the U-shaped fence. Therefore, the position of each object, vehicle and pedestrian in the standard intersection model can be determined.

[0097] like Figure 7 As shown, as a preferred embodiment of the present invention, the pedestrian trajectory prediction module 300 includes:

[0098] The area division unit 301 is used to set a pedestrian detection area in the intersection standard model based on the pedestrian's travel direction, and the pedestrian detection area covers the crosswalk.

[0099] In this module, the area division unit 301 sets a pedestrian detection area in the intersection standard model based on the pedestrian's travel direction. On the road, a crosswalk is set in any travel direction, and the pedestrian detection area is set in the intersection standard model accordingly.

[0100] The image acquisition unit 302 is used to acquire images at a preset acquisition time interval to obtain a binocular image to be detected, wherein the binocular image to be detected includes two groups of real-shot images, and the two groups of real-shot images are respectively from two cameras in the binocular camera.

[0101] In this module, the image acquisition unit 302 performs image acquisition at a preset acquisition time interval, such as once every 50ms, to obtain a set of binocular images to be detected. The binocular camera actually includes two cameras. Therefore, during image acquisition, two sets of real-shot images will be obtained at one time, that is, one camera generates a set of real-shot images.

[0102] The pedestrian recognition unit 303 is used to use a face recognition algorithm to recognize pedestrians in the picture, mark each pedestrian, determine the movement trajectory of the pedestrian according to the continuous binocular images to be detected, generate a movement prediction trajectory, and generate pedestrian information.

[0103] In this module, the pedestrian recognition unit 303 uses a face recognition algorithm to recognize pedestrians in the picture and determines the position of the pedestrians in the picture. In order to distinguish pedestrians, a mark is set for each pedestrian. Specifically, it can be a code, such as pedestrian A00001. When the pedestrian appears in the picture, it is always marked with the code. When the pedestrian no longer appears in the picture, it is given a new code when it enters again. The binocular images to be detected are collected at time intervals. Therefore, each group of binocular images to be detected will determine a position of each pedestrian in the standard model of the intersection once, so the pedestrian can be determined in the standard model of the intersection. The historical movement trajectory of a person can be used to determine the predicted movement trajectory of the pedestrian in the short future time based on the movement trend of the historical movement trajectory. Specifically, a two-dimensional coordinate system can be constructed based on the ground of the standard model of the intersection, so that the historical position of the pedestrian can be marked in the two-dimensional coordinate system. Function fitting is performed according to the coordinate points corresponding to the above historical positions (the horizontal coordinate is the time value, and the vertical coordinate is the distance value between the pedestrian and one end of the zebra crossing) to obtain the pedestrian movement trajectory function. Based on the above pedestrian movement trajectory function, the predicted movement trajectory of the pedestrian in a short time can be determined, and pedestrian information can be generated. The pedestrian information at least includes the height of the pedestrian and the position information of the pedestrian in the horizontal plane.

[0104] like Figure 8 As shown, as a preferred embodiment of the present invention, the safety verification module 400 includes:

[0105] The vehicle identification unit 401 is used to retrieve the binocular image to be detected collected by the binocular camera, identify the vehicle contained therein, and generate vehicle information, wherein the vehicle information at least includes the vehicle type, vehicle size, and driving position height.

[0106] In this module, the vehicle identification unit 401 retrieves the binocular image to be detected collected by the binocular camera. In the binocular image to be detected, not only pedestrians are included, but also vehicles in each lane. Similarly, the image recognition technology is used to identify the license plate number, vehicle size and vehicle type of each vehicle in the picture. The binocular camera can simulate the stereoscopic vision principle of the human eye and use two cameras to capture images of the same scene from different positions. The depth information of the object in the image, that is, the distance from the camera, can be calculated. After obtaining the depth information of the object and the vertical height from the camera to the ground, combined with the geometric relationship, the depth information of the object can be calculated. The height or width of the object or task in the image, such as the size information. Taking the height calculation of a person as an example, obtain the depth information: first, by matching the left and right images captured by the binocular camera, use the triangulation method to calculate the object depth (i.e., the Z-axis distance) corresponding to each pixel in the image; determine the camera height: obtain the precise height of the binocular camera relative to the ground; measure the pixel coordinates of the top of the head and feet of the person: identify and mark the top of the head and feet of the person in the image, and obtain the pixel coordinates of these two points in the image; calculate the actual height: using the principle of similar triangles, the height of the person (H) can be estimated by the following formula:

[0107]

[0108] Among them, h 0 h is the height of the binocular camera minus the vertical distance from the bottom of the camera to the ground. p is the pixel difference from the feet to the top of the head in the image, l s It is the pixel distance corresponding to the depth value from the camera to the plane where the person is standing. Based on this, the height of the person can be calculated. Based on the same principle, the width of the person, the height and width of the vehicle can be calculated. Based on the position of the face in the picture, the position of the driving seat can be determined, thereby obtaining vehicle information and pedestrian information.

[0109] The trajectory verification unit 402 is used to determine the positions of pedestrians and vehicles in the standard model of the intersection based on the vehicle information and pedestrian information, and determine the movement trajectories of pedestrians and vehicles, complete the line of sight occlusion verification and trajectory verification, and obtain the verification results.

[0110] The risk warning unit 403 is used to issue a warning message when it is determined based on the verification result that there is a line of sight obstruction between the vehicle and the pedestrian and the trajectories of the two overlap.

[0111] In this module, the trajectory verification unit 402 determines the positions of pedestrians and vehicles in the intersection standard model based on vehicle information and pedestrian information. Specifically, the intersection standard model is a three-dimensional model, so a three-dimensional coordinate system can be constructed based on the intersection standard model, and the predicted motion trajectory of the pedestrian and the predicted motion trajectory of the vehicle have been determined. Then, the trajectory overlap judgment is first performed, and two trajectories are drawn in the three-dimensional coordinate system to determine the positions of pedestrians and vehicles at each moment. If at a certain moment, the distance between the positions of the pedestrians and the vehicles is less than a preset value, it is determined that there is a trajectory overlap, and then the line of sight occlusion verification is started. The position of the driving seat is determined based on the vehicle information, and the coordinates of the driving seat are determined. The steps include: image acquisition: binocular cameras simultaneously shoot the same scene from slightly different perspectives; parallax calculation: by comparing the pixel position difference (parallax) of the same object in the images captured by the two cameras, the distance of the object from the camera is known. The larger the parallax, the closer the object is to the camera; the smaller the parallax, the farther the object is; three-dimensional reconstruction: Using parallax information and the known baseline distance between cameras (the distance between the centers of the two cameras), combined with the internal parameters of the camera (such as focal length) and external parameters (such as the position and orientation of the camera), the coordinates of the object corresponding to each pixel in the image in three-dimensional space are calculated through triangulation; Depth information and position calculation: Once the depth information of each pixel is obtained, the three-dimensional structure of the entire scene can be reconstructed, including the spatial position of the object (X, Y, Z coordinates). In the same way, the coordinates of the pedestrian are determined. In order to ensure reliability, the coordinates at half the height of the pedestrian are used as the coordinates of the pedestrian. A line is drawn based on the coordinates of the pedestrian and the coordinates of the driver's seat. If the line between the two is blocked by an obstacle, it is determined that there is a line of sight obstruction. Otherwise, it is determined that there is no line of sight obstruction. When it is determined that there is both a line of sight obstruction and trajectory overlap, an alarm is issued. The alarm method can be to warn the vehicle and pedestrians through sound and light signals, or to temporarily switch the green light to a flashing yellow light to give a high-intensity warning to the vehicle, so as to prompt the vehicle to slow down and avoid.

[0112] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0114] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A pedestrian detection method based on computer vision, characterized in that: The method comprises: Constructing a basic intersection model, wherein the basic intersection model at least includes a road model, a building model, and a device model, wherein the device model at least includes a model of a binocular camera; The binocular camera is used to collect images, and the distance is calculated for the collected binocular ranging images to determine the location of each object in the picture and generate a standard intersection model; A pedestrian detection area is defined in the standard intersection model, and pedestrians in the pedestrian detection area are identified by a binocular camera, each pedestrian is marked, and the predicted motion trajectory of each pedestrian is calculated; Each vehicle is identified through a binocular camera, the predicted motion trajectory of the vehicle is calculated, line of sight obstruction verification and trajectory verification are performed based on the standard intersection model, and warning information is issued based on the verification results.

2. The method for pedestrian detection based on computer vision according to claim 1, characterized in that: The steps of collecting images through a binocular camera, measuring distances on the collected binocular ranging images, determining the positions of various objects in the image, and generating a standard intersection model specifically include: The image to be measured is acquired through the binocular camera, and each object in the image to be measured is identified; The reference distance value between each object and the fixed reference object in the intersection is calculated using the binocular ranging algorithm; The basic intersection model is retrieved, and the dynamic models of various objects are generated in the basic intersection model based on the reference distance value to obtain the standard intersection model.

3. The method for pedestrian detection based on computer vision according to claim 1, characterized in that: The steps of defining a pedestrian detection area in the standard intersection model, identifying pedestrians in the pedestrian detection area by using a binocular camera, marking each pedestrian, and calculating the predicted motion trajectory of each pedestrian specifically include: Based on the pedestrian's travel direction, a pedestrian detection area is set in the intersection standard model, and the pedestrian detection area covers the crosswalk; Perform image acquisition at a preset acquisition time interval to obtain a binocular image to be detected, wherein the binocular image to be detected includes two groups of real-shot images, and the two groups of real-shot images are respectively from two cameras in the binocular camera; The face recognition algorithm is used to identify pedestrians in the picture, mark each pedestrian, determine the pedestrian's motion trajectory based on the continuous binocular images to be detected, generate a motion prediction trajectory, and generate pedestrian information.

4. The method for pedestrian detection based on computer vision according to claim 3, characterized in that: The steps of identifying each vehicle by a binocular camera, calculating the predicted motion trajectory of the vehicle, performing sight blockage verification and trajectory verification based on the intersection standard model, and issuing a warning message based on the verification result specifically include: Retrieving a binocular image to be detected acquired by a binocular camera, identifying the vehicle contained therein, and generating vehicle information, wherein the vehicle information includes at least the vehicle type, vehicle size, and driving position height; Based on vehicle information and pedestrian information, the positions of pedestrians and vehicles in the standard model of the intersection are determined, and the movement trajectories of pedestrians and vehicles are determined, and the line of sight occlusion verification and trajectory verification are completed to obtain the verification results; When it is determined based on the verification results that there is a visual obstruction between the vehicle and the pedestrian and their trajectories overlap, a warning message is issued.

5. The method for pedestrian detection based on computer vision according to claim 1, characterized in that: The step of issuing a warning message includes sending a control signal to a traffic light to warn the vehicle via the traffic light.

6. A pedestrian detection system based on computer vision, characterized in that: The system comprises: A basic model building module, used to build a basic intersection model, wherein the basic intersection model includes at least a road model, a building model and a device model, wherein the device model includes at least a model of a binocular camera; The standard model generation module is used to collect images through a binocular camera, measure the distance of the collected binocular ranging images, determine the position of each object in the picture, and generate a standard model of the intersection; The pedestrian trajectory prediction module is used to define the pedestrian detection area in the standard intersection model, identify pedestrians in the pedestrian detection area through a binocular camera, mark each pedestrian, and calculate the predicted motion trajectory of each pedestrian; The safety verification module is used to identify each vehicle through a binocular camera, calculate the vehicle's predicted motion trajectory, perform line of sight occlusion verification and trajectory verification based on the standard intersection model, and issue warning information based on the verification results.

7. The computer vision-based pedestrian detection system according to claim 6, characterized in that: The standard model generation module includes: An object recognition unit is used to acquire images to be measured and calculated by using a binocular camera, and to recognize each object in the image to be measured and calculated; A distance value calculation unit, used to calculate the reference distance value between each object and a fixed reference object in the intersection by using a binocular distance measurement algorithm; The standard model generation unit is used to retrieve the basic intersection model, generate dynamic models of various objects in the basic intersection model based on the reference distance value, and obtain the standard intersection model.

8. The computer vision-based pedestrian detection system according to claim 6, characterized in that: The pedestrian trajectory prediction module includes: A region division unit, configured to set a pedestrian detection region in a standard intersection model based on a pedestrian's travel direction, wherein the pedestrian detection region covers a crosswalk; An image acquisition unit is used to acquire images at a preset acquisition time interval to obtain a binocular image to be detected, wherein the binocular image to be detected includes two groups of real-shot images, and the two groups of real-shot images are respectively from two cameras in the binocular camera; The pedestrian recognition unit is used to use the face recognition algorithm to identify pedestrians in the picture, mark each pedestrian, determine the movement trajectory of the pedestrian based on the continuous binocular images to be detected, generate the movement prediction trajectory, and generate pedestrian information.

9. The computer vision-based pedestrian detection system according to claim 8, characterized in that: The safety verification module comprises: A vehicle identification unit is used to retrieve the binocular image to be detected collected by the binocular camera, identify the vehicle contained therein, and generate vehicle information, wherein the vehicle information at least includes the vehicle type, vehicle size, and driving position height; A trajectory verification unit is used to determine the positions of pedestrians and vehicles in the standard model of the intersection based on vehicle information and pedestrian information, and to determine the movement trajectories of pedestrians and vehicles, complete line of sight occlusion verification and trajectory verification, and obtain verification results; The risk warning unit is used to issue a warning message when it is determined based on the verification result that there is a visual obstruction between the vehicle and the pedestrian and their trajectories overlap.

10. The computer vision-based pedestrian detection system according to claim 6, characterized in that: The step of issuing a warning message includes sending a control signal to a traffic light to warn the vehicle via the traffic light.

Images