Pedestrian recognition method on a crosswalk

Abstract

The application discloses a kind of identification methods of pedestrian on crosswalk, the method utilizes vehicle front camera to obtain the image corresponding to vehicle front scene, obtains the color block in image and carries out screening, and then obtains the regression straight line of color block, utilizes the characteristics of the longitudinal boundary line of white color block of crosswalk in the image of front camera compared to a point to the regression straight line in image is screened and numbered, and the regression straight line screened is the longitudinal boundary line of white color block of crosswalk.By the change of regression straight line number in detection process, the shielding condition of pedestrian to the white color block of crosswalk is reflected, and then the identification of pedestrian on crosswalk is realized.

Description

Technical Field

[0001] This invention relates to the field of vehicle image processing technology, specifically to a method for recognizing pedestrians on crosswalks. Background Technology

[0002] This invention relates to the field of vehicle perception and detection technology, specifically to the application and implementation of a method for pedestrian recognition and estimation at crosswalks. Currently, related autonomous driving systems generally include a millimeter-wave radar and a dedicated ADAS camera (with a built-in intelligent driving vision chip) to meet the needs of intelligent driving. However, millimeter-wave radar and dedicated ADAS cameras are very expensive and are generally used for advanced and comfort functions such as ACC adaptive cruise control and TJA traffic jam assist. Adding a millimeter-wave radar or a dedicated ADAS camera to a vehicle that only requires basic functions like AEB would incur additional costs. Summary of the Invention

[0003] The purpose of this invention is to provide a method for identifying pedestrians on crosswalks, which uses a vehicle's forward-facing camera for pedestrian identification, thereby reducing the hardware costs incurred in vehicle development to meet certain driver assistance functions.

[0004] To solve the above-mentioned technical problems, the present invention provides a technical solution: a method for identifying pedestrians on crosswalks, the method comprising the following steps:

[0005] Using a calibrated forward-facing camera, key feature points of color blocks in the acquired image are identified. A certain range is then expanded outward from the key feature points as the center, and this expanded range is used as the initial candidate range for the color blocks. The key feature points include the corner points and boundary points of the color blocks.

[0006] Within the initial candidate range of color blocks, points with adjacent features are identified and several sets are formed. The sets are then filtered based on the pixel length of all sets.

[0007] The dividing lines of the color blocks in the pedestrian crossing are determined to intersect at a vanishing point in the image captured by the forward-facing camera;

[0008] Obtain the regression lines of the filtered set, group all regression lines into pairs to form several coarse judgment groups, and determine whether each group of regression lines meets the coarse judgment requirements of vanishing point based on the two-dimensional coordinates of the regression points on the regression lines of each coarse judgment group and the intersection of the two regression lines in the image plane.

[0009] The coarse judgment groups that meet the coarse judgment requirements of vanishing point are grouped into pairs to form several fine judgment groups. Based on the spherical coordinates of the intersection of the regression lines of each fine judgment group, it is determined whether each fine judgment group meets the fine judgment requirements of vanishing point.

[0010] Number the regression lines that meet the accuracy requirements;

[0011] The presence of pedestrians on the crosswalk ahead is determined by the changes in the number of the regression line during real-time detection.

[0012] According to the above scheme, the forward-facing camera is calibrated in the following way: the transformation relationship between the pixel coordinates of the image obtained by the forward-facing camera and the actual three-dimensional coordinates of the scene in front of the vehicle is established based on the extrinsic and intrinsic parameters of the vehicle-mounted forward-facing camera; then the vehicle is placed in the test scene, and the forward-facing camera is calibrated by determining the pixel coordinates of the test pattern in the image obtained by the forward-facing camera and the actual three-dimensional coordinates in the test scene.

[0013] According to the above scheme, the process of establishing the transformation relationship between the pixel coordinates of the image obtained by the forward-facing camera and the actual three-dimensional coordinates of the scene in front of the vehicle is as follows.

[0014] Let [XYZ] T [uv] represents the actual 3D coordinates of the scene in front of the vehicle. T Here are the pixel coordinates of the image captured by the front-facing camera. The transformation relationship between the two coordinates mentioned above is as follows:

[0015]

[0016] Where ρ is a constant, R and t are extrinsic parameters of the front-facing camera, f is the focal length of the front-facing camera, (u0, v0) are the pixel coordinates of the optical axis of the front-facing camera in the image, and (dx, dy) are the resolution of the camera in the X and Y directions; m ij Let i be a constant, i = 1, 2, 3; j = 1, 2, 3, 4;

[0017] Since this method is designed for pedestrian crossings, it obtains the following by setting the Z-direction to 0:

[0018]

[0019] Simplifying the above equation further, we get:

[0020]

[0021] According to the above scheme, the test pattern includes n key feature points. The two-dimensional coordinates of each key feature point in the scene in front of the vehicle are obtained through calibration (X...). k Y k ), whose pixel coordinates in the image are (u k v k ); where k = 1, 2, 3, ..., n; therefore, we have:

[0022]

[0023] According to the above scheme, the specific method for obtaining the initial candidate range of color blocks is as follows:

[0024] If the key feature points in the color block are represented by vectors as (p, θ), then the initial candidate range is represented as (p...). s ,θ s ), where p s = [p-Δp, p+Δp], θ s = [θ-Δθ, θ+Δθ], where Δp and Δθ are the set extended range values.

[0025] According to the above scheme, the method for filtering sets based on the pixel length of all sets is as follows: obtain the pixel length H of each set, and calculate the average pixel length of all sets. Calculate the pixel length evaluation value H' for all sets, and retain the set whose pixel length evaluation value H' is greater than its set threshold; where

[0026]

[0027] According to the above scheme, the actual three-dimensional coordinates of the intersection point P of the two regression lines in the same coarse judgment group in the scene in front of the vehicle are set as follows: In the actual 3D coordinate system, the Z-axis is parallel to the optical axis of the front-facing camera, the X-axis is horizontal, and the Y-axis is vertical; s is the distance from the front-facing camera to the point where the optical axis of the front-facing camera falls.

[0028] Let the regression points on the two regression lines A and B in a coarse judgment group be A1(x1,y1), A2(x2,y2), B1(x3,y3), and B2(x4,y4), respectively; define t as the coarse judgment coefficient, then we have:

[0029]

[0030] Therefore:

[0031] t=(y2-y1)*(x4-x3)-(y4-y3)*(x2-x1)

[0032] If t is less than its set threshold, it means that the regression lines A and B meet the coarse criteria for vanishing point judgment.

[0033] According to the above scheme, let the spherical coordinates of the intersection point P of the regression lines be (α, β), where α and β satisfy:

[0034]

[0035]

[0036] Define δ as the precision judgment coefficient, and δ satisfies:

[0037] δ=arccos(cos(90-β1)cos(90-β2)+sin(90-β1)sin(90-β2)cos(α1-α2))

[0038] Where (α1, β1) and (α2, β2) are the spherical coordinates of the intersection of the regression lines in the precision judgment group to be judged;

[0039] If δ is less than its judgment threshold, then the regression line in the corresponding fine judgment group satisfies the vanishing point fine judgment requirement.

[0040] A vehicle active braking method is provided, which uses the pedestrian identification method on crosswalks described above to identify pedestrians. When a pedestrian is identified and the vehicle speed is greater than a certain threshold, the vehicle actively brakes.

[0041] A pedestrian identification system for crosswalks, the system being used to execute the pedestrian identification method for crosswalks described above, the system comprising,

[0042] Forward-facing camera; used to acquire corresponding images of the scene in front of the vehicle;

[0043] Pedestrian detection module; used to identify pedestrians on crosswalks in the scene in front of the vehicle based on the calibration results of the forward-facing camera and the images obtained by the forward-facing camera.

[0044] The beneficial effects of this invention are as follows: In the identification stage, based on the consistent size of the white blocks in the crosswalk, the set of points is initially screened by the length of the blocks. Then, utilizing the parallel longitudinal boundaries of the white blocks in the crosswalk, vanishing points are defined, and coarse and fine criteria for vanishing point judgment are specified. The regression lines of the set are then screened, and the resulting regression lines represent the longitudinal boundary lines of the white blocks in the crosswalk. By numbering these regression lines, changes in the numbers can reflect the occlusion of white blocks by pedestrians in the image, thus identifying pedestrians. This method utilizes only a forward-facing camera, typically found in low-end vehicles, for image acquisition, significantly reducing hardware costs compared to existing technologies that use millimeter-wave radar and dedicated ADAS cameras for pedestrian recognition. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of a test scenario used for camera calibration in existing technologies;

[0046] Figure 2 This is a schematic flowchart of the pedestrian identification method on a crosswalk according to Embodiment 1 of the present invention;

[0047] Figure 3 The image is obtained by a forward-facing camera in a test scenario during the calibration phase of the pedestrian identification method at a crosswalk according to Embodiment 1 of the present invention.

[0048] Figure 4 The pedestrian identification method at a crosswalk according to Embodiment 1 of the present invention is shown in the image after preliminary processing of a real-world image;

[0049] Figure 5 The pedestrian identification method on crosswalks in Embodiment 1 of the present invention is based on a depth-processed image of a real-world scene. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0051] Example 1:

[0052] See Figure 2 A method for identifying pedestrians on crosswalks, comprising a calibration stage and an identification stage;

[0053] During the calibration phase, the transformation relationship between the pixel coordinates of the image obtained by the forward-facing camera and the actual three-dimensional coordinates of the scene in front of the vehicle is established based on the extrinsic and intrinsic parameters of the forward-facing camera. Then, the vehicle is placed in the test scene, and the forward-facing camera is calibrated by determining the pixel coordinates of the test pattern in the image obtained by the forward-facing camera and the actual three-dimensional coordinates in the test scene.

[0054] The identification phase includes the following steps;

[0055] Using a forward-facing camera, key feature points of color blocks in the acquired image are identified (the original image has undergone simple grayscale processing and edge feathering; see the processed image). Figure 4 This allows the color blocks in the image to represent key feature points. Using these key feature points as the center, a certain range is expanded outwards, and this expanded range is used as the initial candidate range for the color blocks (see the image obtained). Figure 5 The image features selected through the initial candidate range are clear and have little noise; among them, key feature points include corner points and boundary points.

[0056] Within the initial candidate range of color blocks, points with adjacent features are identified and several sets are formed. The sets are then filtered based on the pixel length of all sets.

[0057] The dividing lines of the color blocks in the pedestrian crossing are determined to intersect at a vanishing point in the image captured by the forward-facing camera;

[0058] Obtain the regression lines of the filtered set, group all regression lines into pairs to form several coarse judgment groups, and determine whether each group of regression lines meets the coarse judgment requirements of vanishing point based on the two-dimensional coordinates of the regression points on the regression lines of each coarse judgment group and the intersection of the two regression lines in the image plane.

[0059] The coarse judgment groups that meet the coarse judgment requirements of vanishing point are grouped into pairs to form several fine judgment groups. Based on the spherical coordinates of the intersection of the regression lines of each fine judgment group, it is determined whether each fine judgment group meets the fine judgment requirements of vanishing point.

[0060] Number the regression lines that meet the accuracy requirements;

[0061] The presence of pedestrians on the crosswalk ahead is determined by the changes in the number of the regression line during real-time detection.

[0062] Furthermore, during the calibration phase, the process of establishing the transformation relationship between the pixel coordinates of the image acquired by the forward-facing camera and the actual three-dimensional coordinates of the scene in front of the vehicle is as follows:

[0063] Let [XYZ] T [uv] represents the actual 3D coordinates of the scene in front of the vehicle. T Here are the pixel coordinates of the image captured by the front-facing camera. The transformation relationship between the two coordinates mentioned above is as follows:

[0064]

[0065] Where ρ is a constant, R and t are extrinsic parameters of the front-facing camera, f is the focal length of the front-facing camera, (u0, v0) are the pixel coordinates of the optical axis of the front-facing camera in the image, and (dx, dy) are the resolution of the camera in the X and Y directions; m ij Let i be a constant, i = 1, 2, 3; j = 1, 2, 3, 4;

[0066] R and t are the coordinate transformation matrices corresponding to the vehicle, and their specific standard settings are as follows:

[0067] 1) The i-th row of R represents the coordinates of the unit vector in the i-th coordinate axis direction of the forward camera coordinate system in the world coordinate system;

[0068] 2) The i-th column of R represents the coordinates of the i-th coordinate axis in the world coordinate system in the forward camera coordinate system;

[0069] 3) t represents the coordinates of the far point in the world coordinate system in the forward camera coordinate system.

[0070] Since this method is designed for pedestrian crossings, it obtains the following by setting the Z-direction to 0:

[0071]

[0072] Simplifying the above equation further, we get:

[0073]

[0074] Furthermore, in the calibration phase, the test pattern includes n key feature points. The two-dimensional coordinates of each key feature point in the scene in front of the vehicle are obtained through calibration (X...). k Y k ), whose pixel coordinates in the image are (u k v k ); where k = 1, 2, 3, ..., n; therefore, we have:

[0075]

[0076] See Figure 1 Existing technical solutions require a significant amount of time for camera calibration; see [link / reference] Figure 3 In this embodiment, in order to achieve rapid calibration, the test pattern on the ground is arranged in a grid. During calibration, the horizontal straight line that makes up the ground grid in the image acquired by the forward camera is parallel to the bottom edge of the image of the forward camera, and the optical axis of the forward camera is parallel to the vertical straight line of the ground grid. Then, the coordinates of the calibration point on the ground are measured in one go.

[0077] Figure 3 This is a standard diagram used in the test scenario during the calibration phase. When the vehicle is off-line tested, it is calibrated using this diagram and used to write correction parameters. Compared with the calibration process of the prior art, the calibration process of this embodiment is the same as the prior art except that the calibration diagram is different from the prior art.

[0078] Furthermore, the specific method for obtaining the initial candidate range of color blocks is as follows:

[0079] If the key feature points in the color block are represented by vectors as (p, θ), then the initial candidate range is represented as (p...). s ,θ s ), where p s = [p-Δp, p+Δp], θ s = [θ-Δθ, θ+Δθ], where Δp and Δθ are the set extended range values.

[0080] Furthermore, the method for filtering sets based on the pixel length of all sets specifically involves obtaining the pixel length H of each set and calculating the average pixel length of all sets. Calculate the pixel length evaluation value H' for all sets, and retain the set where the pixel length evaluation value H' is greater than a set threshold (in this embodiment, the set threshold corresponding to H' is 25%); where

[0081] Furthermore, the actual three-dimensional coordinates of the intersection point P of the two regression lines in the same coarse judgment group in the scene in front of the vehicle are set as follows: In the actual 3D coordinate system, the Z-axis is parallel to the optical axis of the front-facing camera, the X-axis is horizontal, and the Y-axis is vertical; s is the distance from the front-facing camera to the point where the optical axis of the front-facing camera falls.

[0082] Let the regression points on the two regression lines A and B in a coarse judgment group be A1(x1,y1), A2(x2,y2), B1(x3,y3), and B2(x4,y4), respectively; define t as the coarse judgment coefficient, then we have:

[0083]

[0084] Therefore:

[0085] t=(y2-y1)*(x4-x3)-(y4-y3)*(x2-x1)

[0086] If t is less than its set threshold, it means that the regression lines A and B meet the coarse criteria for vanishing point judgment.

[0087] Furthermore, let the spherical coordinates of the intersection point P of the regression lines be (α, β), where α and β satisfy:

[0088]

[0089]

[0090] Define δ as the precision judgment coefficient, and δ satisfies:

[0091] δ=arccos(cos(90-β1)cos(90-β2)+sin(90-β1)sin(90-β2)cos(α1-α2))

[0092] Where (α1, β1) and (α2, β2) are the spherical coordinates of the intersection of the regression lines in the precision judgment group to be judged;

[0093] If δ is less than its judgment threshold, then the regression line in the corresponding fine judgment group satisfies the vanishing point fine judgment requirement.

[0094] A vehicle active braking method is provided, which uses the pedestrian identification method on crosswalks described above to identify pedestrians. When a pedestrian is identified and the vehicle speed is greater than a certain threshold, the vehicle actively brakes.

[0095] A pedestrian identification system for crosswalks, the system being used to execute the pedestrian identification method for crosswalks described above, the system comprising,

[0096] Forward-facing camera; used to acquire corresponding images of the scene in front of the vehicle;

[0097] Pedestrian detection module; used to identify pedestrians on crosswalks in the scene in front of the vehicle based on the calibration results of the forward-facing camera and the images obtained by the forward-facing camera.

[0098] A vehicle with the pedestrian recognition system described above for crosswalks.

[0099] Example 2:

[0100] The difference between this embodiment and Embodiment 1 is that, in the identification stage, the regression line number is compared with a preset identification value to perform identification. For example, if the regression line number changes as 12356→1236→126 (occlusion type); or as 12456→12356→12346 (deviation type), then it is determined that there is a pedestrian on the crosswalk in front of the vehicle.

[0101] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for identifying pedestrians on a crosswalk, characterized in that: The method includes the following steps: Using a calibrated forward-facing camera, key feature points of color blocks in the acquired image are identified. A certain range is then expanded outward from the key feature points as the center, and this expanded range is used as the initial candidate range for the color blocks. The key feature points include the corner points and boundary points of the color blocks. Within the initial candidate range of color blocks, points with adjacent features are identified and several sets are formed. The sets are then filtered based on the pixel length of all sets. The dividing lines of the color blocks in the pedestrian crossing are determined to intersect at a vanishing point in the image captured by the forward-facing camera; Obtain the regression lines of the filtered set, group all regression lines into pairs to form several coarse judgment groups, and determine whether each group of regression lines meets the coarse judgment requirements of vanishing point based on the two-dimensional coordinates of the regression points on the regression lines of each coarse judgment group and the intersection of the two regression lines in the image plane. The coarse judgment groups that meet the coarse judgment requirements of vanishing point are grouped into pairs to form several fine judgment groups. Based on the spherical coordinates of the intersection of the regression lines of each fine judgment group, it is determined whether each fine judgment group meets the fine judgment requirements of vanishing point. Number the regression lines that meet the accuracy requirements; The presence of pedestrians on the crosswalk ahead is determined by the changes in the number of the regression line during real-time detection.

2. The method for identifying pedestrians at crosswalks according to claim 1, characterized in that: The forward-facing camera is calibrated in the following way: the transformation relationship between the pixel coordinates of the image obtained by the forward-facing camera and the actual three-dimensional coordinates of the scene in front of the vehicle is established based on the extrinsic and intrinsic parameters of the forward-facing camera; then the vehicle is placed in the test scene, and the forward-facing camera is calibrated by determining the pixel coordinates of the test pattern in the image obtained by the forward-facing camera and the actual three-dimensional coordinates in the test scene.

3. The method for identifying pedestrians at crosswalks according to claim 2, characterized in that: The process of establishing the transformation relationship between the pixel coordinates of the image captured by the forward-facing camera and the actual three-dimensional coordinates of the scene in front of the vehicle is as follows. Let [XYZ] T [uv] represents the actual 3D coordinates of the scene in front of the vehicle. T Here are the pixel coordinates of the image captured by the front-facing camera. The transformation relationship between the two coordinates mentioned above is as follows: Where ρ is a constant, R and t are extrinsic parameters of the front-facing camera, f is the focal length of the front-facing camera, (u0, v0) are the pixel coordinates of the optical axis of the front-facing camera in the image, and (dx, dy) are the resolution of the camera in the X and Y directions; m ij Let i be a constant, i = 1, 2, 3; j = 1, 2, 3, 4; Since the identification is for pedestrian crossings, it is obtained by setting the Z direction to 0: Simplifying the above equation further, we get:

4. The method for identifying pedestrians on crosswalks according to claim 3, characterized in that: During the calibration phase, the test pattern includes n key feature points. The two-dimensional coordinates of each key feature point in the scene in front of the vehicle are obtained through calibration (X...). k Y k ), whose pixel coordinates in the image are (u k v k ); where k = 1, 2, 3, ..., n; therefore, we have:

5. The method for identifying pedestrians at crosswalks according to claim 1, characterized in that: The specific method for obtaining the initial candidate range of color blocks is as follows: If the key feature points in the color block are represented by vectors as (p, θ), then the initial candidate range is represented as (p...). s ,θ s ), where p s = [p-Δp, p+Δp], θ s = [θ-Δθ, θ+Δθ], where Δp and Δθ are the set extended range values.

6. The method for identifying pedestrians at crosswalks according to claim 1, characterized in that: The method for filtering sets based on the pixel length of all sets is as follows: obtain the pixel length H of each set, and calculate the average pixel length of all sets. Calculate the pixel length evaluation value H' for all sets, and retain the set whose pixel length evaluation value H' is greater than its set threshold; where 7. The method for identifying pedestrians at crosswalks according to claim 1, characterized in that: Let the actual 3D coordinates of the intersection point P of two regression lines in the same coarse judgment group in the scene in front of the vehicle be defined as follows: In the actual 3D coordinate system, the Z-axis is parallel to the optical axis of the front-facing camera, the X-axis is horizontal, and the Y-axis is vertical; s is the distance from the front-facing camera to the point where the optical axis of the front-facing camera falls. Let the regression points on the two regression lines A and B in a coarse judgment group be A1(x1,y1), A2(x2,y2), B1(x3,y3), and B2(x4,y4), respectively; define t as the coarse judgment coefficient, then we have: Therefore: t=(y2-y1)*(x4-x3)-(y4-y3)*(x2-x1) If t is less than its set threshold, it means that the regression lines A and B meet the coarse criteria for vanishing point judgment.

8. The method for identifying pedestrians at crosswalks according to claim 7, characterized in that: Let the spherical coordinates of the intersection point P of the regression lines be (α, β), where α and β satisfy: Define δ as the precision judgment coefficient, and δ satisfies: δ=arccos(cos(90-β1)cos(90-β2)+sin(90-β1)sin(90-β2)cos(α1-α2)) Where (α1, β1) and (α2, β2) are the spherical coordinates of the intersection of the regression lines in the precision judgment group to be judged; If δ is less than its judgment threshold, then the regression line in the corresponding fine judgment group satisfies the vanishing point fine judgment requirement.

9. A method for active braking of a vehicle, characterized in that: This method uses the pedestrian identification method on crosswalks as described in any one of claims 1-8 to identify pedestrians. When a pedestrian is identified and the vehicle speed is greater than a certain threshold, the vehicle actively brakes.

10. A pedestrian identification system for crosswalks, characterized in that: This system is used to perform the pedestrian identification method on a crosswalk as described in any one of claims 1-8, and the system includes, Forward-facing camera; Used to acquire the corresponding image of the scene in front of the vehicle; Pedestrian detection module; It is used to identify pedestrians on crosswalks in the scene in front of the vehicle based on the calibration results of the forward-facing camera and the images obtained by the forward-facing camera.

Images