Method for recognizing pedestrians on pedestrian crosswalk

Abstract

Disclosed in the present invention is a method for recognizing pedestrians on a pedestrian crosswalk. In the method, an image corresponding to a scene in front of a vehicle is acquired by means of a front-facing camera of a vehicle; color blocks in the image are acquired and screened; regression lines of the color blocks are then acquired; and regression lines in the image are screened and numbered by utilizing the characteristic that vertical boundary lines of white color blocks on a pedestrian crosswalk intersect at a point in the image from the front-facing camera, wherein the successfully screened regression lines are the vertical boundary lines of the white color blocks on the pedestrian crosswalk. During a detection process, the variation in the numbering of regression lines reflects the occlusion of white color blocks on a pedestrian crosswalk caused by pedestrians, thus realizing the recognition of pedestrians on the pedestrian crosswalk.

Description

A method for identifying pedestrians on a crosswalk

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The embodiments of this application are based on the Chinese patent application with application number 202311186786.6 and application date September 12, 2023, and claim the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into the embodiments of this application as a reference. Technical Field

[0003] The present invention relates to the technical field of driving image processing, and in particular to a method for identifying pedestrians on a crosswalk. Background Art

[0004] The present invention relates to the field of vehicle perception and detection technology, and specifically to the application and implementation of a method for identifying and inferring pedestrians at a crosswalk. Currently, related autonomous driving systems are generally equipped with a millimeter-wave radar and a dedicated camera for an ADAS system (with a built-in intelligent driving vision chip) during product development, which has met the needs of intelligent driving of vehicles. However, the cost of millimeter-wave radars or dedicated cameras for ADAS systems is very high, and they are generally used for advanced and comfort functions such as ACC adaptive cruise control and TJA traffic congestion assistance. If a millimeter-wave radar or a dedicated camera for an ADAS system is added to a car model that only requires low-end functions such as an AEB system, it will bring additional costs.

[0005] Summary of the Invention

[0006] The purpose of the present invention is to provide a method for identifying pedestrians on a crosswalk, so as to utilize a vehicle's forward-facing camera to perform pedestrian recognition, thereby reducing the hardware cost incurred in vehicle development to meet some assisted driving functions.

[0007] In order to solve the above technical problems, the present invention provides a technical solution: a method for identifying pedestrians on a crosswalk, which is applied to electronic equipment in a vehicle.

[0008] Using a calibrated forward-facing camera, identify key feature points of a color block in the image, and use the key feature points as the center to expand a certain range outward as a preliminary candidate range for the color block; the key feature points include the corner points and boundary points of the color block;

[0009] In the preliminary candidate range of the color block, points with adjacent features are identified and formed into several sets, and the sets are screened according to the pixel length of all sets;

[0010] Determine that the boundary lines of the color blocks in the crosswalk intersect at a vanishing point in the image obtained by the forward-facing camera;

[0011] Obtain the regression lines of the filtered set, group all the regression lines into pairs to form several rough judgment groups, and determine whether each group of regression lines meets the rough judgment requirements of the vanishing point based on the regression points on the regression lines of each rough judgment group and the two-dimensional coordinates of the intersection of the two regression lines in the image plane;

[0012] The rough judgment groups that meet the rough judgment requirements of the vanishing point are divided into two groups to form several fine judgment groups. Based on the spherical coordinates of the intersection points of the regression lines of each fine judgment group, it is judged whether each fine judgment group meets the vanishing point fine judgment requirements;

[0013] Number the regression lines that meet the requirements of precise judgment;

[0014] The presence of pedestrians on the crosswalk ahead is determined based on the change in the number of the regression line during real-time detection.

[0015] As a second aspect, the present invention also provides a vehicle active braking method, which uses the pedestrian recognition method on the crosswalk described above to identify pedestrians. When a pedestrian is recognized and the vehicle speed is greater than a certain threshold, the vehicle actively brakes.

[0016] As a third aspect, the present invention further provides a system for identifying pedestrians on a crosswalk, the system being used to execute the above-mentioned method for identifying pedestrians on a crosswalk, the system comprising:

[0017] Forward-facing camera; used to obtain corresponding images of the scene in front of the vehicle;

[0018] Pedestrian detection module; used to identify pedestrians on the crosswalk in the scene in front of the vehicle based on the calibration results of the forward-facing camera and the image obtained by the forward-facing camera.

[0019] As a fourth aspect, the present invention further provides a car comprising the aforementioned pedestrian recognition system on a crosswalk.

[0020] The beneficial effects of the present invention are as follows: During the recognition phase of the method, based on the consistent actual size of the white blocks in the crosswalk, a preliminary screening of the set of points is performed using the block length; then, utilizing the parallel longitudinal boundaries of the white blocks in the crosswalk, vanishing points are defined and rough and fine vanishing point determination requirements are specified, and the regression line of the set is screened. The resulting regression line is the longitudinal boundary line of the white blocks in the crosswalk. By numbering the screened regression lines, the changes in the numbers can reflect the extent to which pedestrians in the image obstruct the white blocks in the crosswalk, thereby identifying pedestrians. This method utilizes only the forward-facing camera, which is commonly found in low-end vehicles, for image acquisition, significantly reducing hardware costs compared to existing solutions that utilize millimeter-wave radar and ADAS-specific cameras for pedestrian recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG1 is a schematic diagram of a test scene used for camera calibration in the prior art;

[0022] FIG2 is a flow chart of a method for identifying pedestrians on a crosswalk according to a first embodiment of the present invention;

[0023] FIG3 is an image captured by a forward-facing camera in a test scene during the calibration phase of the method for identifying pedestrians on a crosswalk according to the first embodiment of the present invention;

[0024] FIG4 is a diagram of a method for identifying pedestrians on a crosswalk according to the first embodiment of the present invention, showing a real scene image after preliminary processing;

[0025] FIG5 is a diagram of a method for identifying pedestrians on a crosswalk according to the first embodiment of the present invention, wherein the real scene image is subjected to depth processing. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0027] Example 1:

[0028] Referring to FIG2 , a method for identifying pedestrians on a crosswalk includes a calibration phase and an identification phase;

[0029] During the calibration phase, a transformation relationship is established between the pixel coordinates of the image captured by the forward-facing camera and the actual 3D coordinates of the scene in front of the vehicle based on the extrinsic and intrinsic parameters of the vehicle's forward-facing camera. The vehicle is then placed in a test scene, and the forward-facing camera is calibrated by determining the pixel coordinates of the test pattern in the test scene in the image captured by the forward-facing camera and its actual 3D coordinates in the test scene.

[0030] The identification phase includes the following steps;

[0031] The forward-facing camera is used to identify the key feature points of the color blocks in the obtained image (the original image is processed with simple grayscale and feathered edges, and the processed image is shown in Figure 4, so that the color blocks in the image can reflect the key feature points). With the key feature points as the center, a certain range is expanded outward, and the expanded range is used as the preliminary candidate range of the color block (the obtained image is shown in Figure 5, and the image feature points screened by the preliminary candidate range are clear and have little noise); the key feature points include corner points and dividing points;

[0032] In the preliminary candidate range of the color block, points with adjacent features are identified and formed into several sets, and the sets are screened according to the pixel length of all sets;

[0033] Determine that the boundary lines of the color blocks in the crosswalk intersect at a vanishing point in the image obtained by the forward-facing camera;

[0034] Obtain the regression lines of the filtered set, group all the regression lines into pairs to form several rough judgment groups, and determine whether each group of regression lines meets the rough judgment requirements of the vanishing point based on the regression points on the regression lines of each rough judgment group and the two-dimensional coordinates of the intersection of the two regression lines in the image plane;

[0035] The rough judgment groups that meet the rough judgment requirements of the vanishing point are divided into two groups to form several fine judgment groups. Based on the spherical coordinates of the intersection points of the regression lines of each fine judgment group, it is judged whether each fine judgment group meets the vanishing point fine judgment requirements;

[0036] Number the regression lines that meet the requirements of precise judgment;

[0037] The presence of pedestrians on the crosswalk ahead is determined based on the change in the number of the regression line during real-time detection.

[0038] Furthermore, during the calibration phase, the transformation relationship between the pixel coordinates of the image obtained by the forward-facing camera and the actual three-dimensional coordinates of the scene in front of the vehicle is established as follows:

[0039] Let [XYZ] T is the actual 3D coordinate of the scene in front of the vehicle, [uv] T is the pixel coordinate of the image obtained by the forward camera. The conversion relationship between the two coordinates is as follows:

[0040] Where ρ is a constant, R and t are the external parameters of the forward camera, f is the focal length of the forward camera, (u0, v0) is the pixel coordinate of the point where the optical axis of the forward camera lands in the image, and (dx, dy) is the resolution of the camera in the X and Y directions; m ij is a constant, i=1,2,3; j=1,2,3,4;

[0041] R and t are the coordinate transformation matrices corresponding to the vehicle, and their specific standard settings are as follows:

[0042] 1) The i-th row of R represents the coordinates of the unit vector in the direction of the i-th coordinate axis in the forward camera coordinate system in the world coordinate system;

[0043] 2) The i-th column of R represents the coordinate of the i-th coordinate axis in the world coordinate system in the forward camera coordinate system;

[0044] 3) t represents the coordinates of the far point of the world coordinate system in the forward camera coordinate system.

[0045] Since this method is used to identify pedestrian crossings, the Z direction is set to 0 to obtain:

[0046] Further simplifying the above formula, we get:

[0047] Furthermore, in the calibration phase, the test pattern includes n key feature points, and the two-dimensional coordinates of each key feature point in the scene in front of the vehicle are obtained by calibration as (X k , Y k ), the pixel coordinates in the image are (u k , v k ); where k = 1, 2, 3, ...., n; so:

[0048] Referring to FIG1 , conventional camera calibration takes a significant amount of time. Referring to FIG3 , in this embodiment, to achieve rapid calibration, a test pattern on the ground is formed into a grid. During calibration, the horizontal lines forming the ground grid in the image captured by the forward-facing camera are aligned with the bottom edge of the image of the forward-facing camera, and the corresponding optical axis of the forward-facing camera is aligned with the vertical lines of the ground grid. This allows the coordinates of the calibration points on the ground to be measured in one go.

[0049] Figure 3 shows a standard diagram in a test scenario during the calibration phase. This diagram is used for calibration during vehicle offline testing and for writing correction parameters. Compared to the calibration process in the prior art, the calibration process in this embodiment differs only in the calibration diagram; the remaining calibration steps are consistent with the prior art.

[0050] Furthermore, the method for obtaining the preliminary candidate range of the color block is as follows:

[0051] The key feature points in the color block are expressed as (p,θ) in vector notation, and the preliminary candidate range is expressed as (p s ,θ s ), where p s =[p-Δp,p+Δp],θ s =[θ-Δθ, θ+Δθ], Δp and Δθ are the set extension range values.

[0052] Furthermore, the method for screening the sets according to the pixel lengths of all sets is specifically to obtain the pixel length H of each set and calculate the mean pixel length of all sets Calculate the pixel length evaluation value H' of all sets, and retain the sets whose pixel length evaluation value H' is greater than the set threshold (the set threshold corresponding to H' in this embodiment is 25%);

[0053] Furthermore, the actual three-dimensional coordinates of the intersection point P of the two regression lines in the same rough judgment group in the scene in front of the vehicle are set as In the actual 3D coordinate system, the Z axis is parallel to the optical axis of the forward camera, the X axis is horizontal, and the Y axis is vertical; s is the distance from the forward camera to the point where the optical axis of the forward camera lands;

[0054] Suppose the regression points on the two regression lines A and B in a rough judgment group are A1(x1,y1), A2(x2,y2), B1(x3,y3), and B2(x4,y4) respectively; define t as the rough judgment coefficient, then:

[0055] So we have: t=(y2-y1)*(x4-x3)-(y4-y3)*(x2-x1)

[0056] If t is less than the set threshold, it means that the regression lines A and B meet the requirements for the rough judgment of the vanishing point.

[0057] Furthermore, let the spherical coordinates of the intersection point P of the regression line be (α, β), where α and β satisfy:

[0058] Define δ as the precision coefficient, and δ satisfies: δ=arccos(cos(90-β1)cos(90-β2)+sin(90-β1)sin(90-β2)cos(α1-α2))

[0059] Where (α1, β1) and (α2, β2) are the spherical coordinates of the intersection points of the regression lines in the precise judgment group to be judged;

[0060] If δ is less than its judgment threshold, the regression line in the corresponding precise judgment group meets the requirements for precise vanishing point judgment.

[0061] A vehicle active braking method utilizes the above-mentioned pedestrian recognition method on a crosswalk to recognize pedestrians. When a pedestrian is recognized and the vehicle speed is greater than a certain threshold, the vehicle actively brakes.

[0062] A pedestrian identification system on a crosswalk, the system is used to implement the pedestrian identification method on a crosswalk described above, the system comprising:

[0063] Forward-facing camera; used to obtain corresponding images of the scene in front of the vehicle;

[0064] Pedestrian detection module; used to identify pedestrians on the crosswalk in the scene in front of the vehicle based on the calibration results of the forward-facing camera and the image obtained by the forward-facing camera.

[0065] A car comprises the above-mentioned pedestrian recognition system on a crosswalk.

[0066] Example 2:

[0067] This embodiment differs from the first embodiment in that, during the recognition phase, the regression line number is compared with a preset recognition number to perform recognition. For example, if the regression line number changes from 12356 to 1236 to 126 (occlusion type) or 12456 to 12356 to 12346 (deviation type), it is determined that there is a pedestrian on the crosswalk ahead of the vehicle.

[0068] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for identifying pedestrians on a crosswalk, applied to an electronic device of a vehicle, characterized in that: Using the calibrated forward camera, identify key feature points of the color block in the obtained image, expand a certain range outward with the key feature point as the center, and use the expanded range as the preliminary candidate range of the color block; wherein the key feature point includes the corner point and the boundary point of the color block; In the preliminary candidate range of the color block, points with adjacent features are identified and formed into several sets, and the sets are screened according to the pixel length of all sets; It is determined that the boundary lines of the color blocks in the crosswalk intersect at a vanishing point in the image obtained by the forward camera; Obtain the regression lines of the filtered set, group all the regression lines in pairs to form several rough judgment groups, and judge whether each group of regression lines meets the rough judgment requirements of the vanishing point according to the regression points on the regression lines of each rough judgment group and the two-dimensional coordinates of the intersection of the two regression lines in the image plane; The rough judgment groups that meet the rough judgment requirements of the vanishing point are divided into groups of two each to form a number of fine judgment groups, and each fine judgment group is judged whether it meets the fine judgment requirements of the vanishing point according to the spherical coordinates of the intersection points of the regression lines of each fine judgment group; Number the regression lines that meet the requirements of precise judgment; According to the change of the number of the regression line during real-time detection, it is judged whether there are pedestrians on the crosswalk ahead.

2. The method for identifying pedestrians on a crosswalk according to claim 1, characterized in that: The forward camera is calibrated in the following way: a transformation relationship between the pixel coordinates of the image obtained by the forward camera and the actual three-dimensional coordinates of the scene in front of the vehicle is established based on the external and internal parameters of the vehicle-mounted forward camera; the vehicle is then placed in a test scene, and the forward camera is calibrated by determining the pixel coordinates of the test pattern in the test scene in the image obtained by the forward camera and the actual three-dimensional coordinates in the test scene.

3. The method for identifying pedestrians on a crosswalk according to claim 2, characterized in that: The specific process of establishing the transformation relationship between the pixel coordinates of the image obtained by the forward camera and the actual three-dimensional coordinates of the scene in front of the vehicle is as follows: Let [XYZ] T is the actual 3D coordinate of the scene in front of the vehicle, [uv] T is the pixel coordinate of the image obtained by the forward camera. The conversion relationship between the above two coordinates is as follows: Where ρ is a constant, R and t are the external parameters of the forward camera, f is the focal length of the forward camera, (u0, v0) is the pixel coordinate of the point where the optical axis of the forward camera falls in the image, and (dx, dy) is the resolution of the camera in the X and Y directions; m ij is a constant, i=1,2,3; j=1,2,3,4; Since this method is used to identify pedestrian crossings, the Z direction is set to 0 to obtain: Further simplifying the above formula, we get:

4. The method for identifying pedestrians on a crosswalk according to claim 3, characterized in that: In the calibration phase, the test pattern includes n key feature points. The two-dimensional coordinates of each key feature point in the scene in front of the vehicle are obtained through calibration as (X k , Y k ), the pixel coordinates in the image are (u k , v k ); where k = 1, 2, 3, ...., n; so:

5. The method for identifying pedestrians on a crosswalk according to claim 1, characterized in that: The method for obtaining the preliminary candidate range of the color block is as follows: The key feature points in the color block are expressed as (p, θ) in vector notation, and the preliminary candidate range is expressed as (p s ,θ s ), where p s =[p-Δp,p+Δp],θ s =[θ-Δθ, θ+Δθ], Δp and Δθ are the set extension range values.

6. The method for identifying pedestrians on a crosswalk according to claim 1, characterized in that: The method for filtering the sets according to the pixel lengths of all sets is specifically to obtain the pixel length H of each set and calculate the mean pixel length of all sets. Calculate the pixel length evaluation value H′ of all sets, and retain the sets whose pixel length evaluation value H′ is greater than the set threshold; 7. The method for identifying pedestrians on a crosswalk according to claim 1, characterized in that: The actual three-dimensional coordinates of the intersection point P of the two regression lines in the same rough judgment group in the scene in front of the vehicle are set as In the actual three-dimensional coordinate system, the Z axis is parallel to the optical axis of the forward camera, the X axis is horizontal, and the Y axis is vertical; s is the distance from the forward camera to the point where the optical axis of the forward camera lands; Suppose the regression points on the two regression lines A and B in a rough judgment group are A1(x1,y1), A2(x2,y2), B1(x3,y3), B2(x4,y4) respectively; define t as the rough judgment coefficient, then: So we have: t=(y2-y1)*(x4-x3)-(y4-y3)*(x2-x1) If t is less than the set threshold, it means that the regression lines A and B meet the requirements for the rough judgment of the vanishing point.

8. The method for identifying pedestrians on a crosswalk according to claim 7, characterized in that: Assume that the spherical coordinates of the intersection point P of the regression line are (α, β), where α and β satisfy: Define δ as the precision judgment coefficient, and δ satisfies: δ=arccos(cos(90-β1)cos(90-β2)+sin(90-β1)sin(90-β2)cos(α1-α2)) Where (α1, β1) and (α2, β2) are the spherical coordinates of the intersection points of the regression lines in the precise judgment group to be judged; If δ is less than its judgment threshold, the corresponding regression line in the precise judgment group meets the requirements for precise judgment of the vanishing point.

9. A vehicle active braking method, characterized in that: The method uses the pedestrian recognition method on the crosswalk described in any one of claims 1 to 8 to recognize pedestrians. When a pedestrian is recognized and the vehicle speed is greater than a certain threshold, the vehicle actively brakes.

10. A pedestrian recognition system on a crosswalk, characterized by: The system is used to execute the method for identifying pedestrians on a crosswalk as described in any one of claims 1 to 8, and the system comprises: Front-facing camera; Used to obtain a corresponding image of the scene in front of the vehicle; Pedestrian detection module; Used to identify pedestrians on the crosswalk in the scene in front of the vehicle based on the calibration results of the forward camera and the image obtained by the forward camera.

11. A car, characterized in that: The invention comprises a pedestrian identification system on a crosswalk as described in claim 10.

Images