Fisheye camera ranging method and terminal

Abstract

The application provides a fisheye camera ranging method and a terminal. According to an image captured by a fisheye camera, a coordinate of a target point in a pixel coordinate system and a pre-calibrated coordinate of a principal point of the fisheye camera in the pixel coordinate system are obtained, and an imaging distortion coordinate of the target point in an imaging plane coordinate system is calculated. According to the imaging distortion coordinate of the target point in the imaging plane coordinate system, an imaging non-distortion coordinate of the target point in the imaging plane coordinate system is determined. The imaging non-distortion coordinate of the target point in the imaging plane coordinate system is converted into the pixel coordinate system to obtain a non-distortion target coordinate. According to the target coordinate, pre-calibrated internal parameters and external parameters of the fisheye camera, a coordinate value of the target point in a world coordinate system is solved. According to the coordinate value of the target point in the world coordinate system, a distance between a vehicle and the target point is determined. The application can improve the ranging accuracy of the fisheye camera.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of assisted driving, and particularly relates to a fisheye camera ranging method and a terminal. BACKGROUND

[0002] An assisted driving system can effectively reduce and prevent the influence of human driving errors and reduce the probability of traffic accidents. The assisted driving system uses sensors installed on a vehicle to perceive the surrounding environment, thereby assisting the driver in driving activities.

[0003] BSD (Blind-Spot-Detection, blind spot detection system) is a kind of assisted driving system, and a camera is the main sensor for BSD perception. Since BSD is a short-range perception and low-speed braking, a fisheye camera with a 160-degree field of view is selected. However, the fisheye camera has a large distortion, and the target ranging is inaccurate in short-range perception, thereby causing misjudgment and misbraking during braking and bringing a poor experience to the driver. SUMMARY

[0004] The present application provides a fisheye camera ranging method and a terminal to solve the problem of inaccurate fisheye camera ranging.

[0005] In a first aspect, the present application provides a fisheye camera ranging method, comprising: fixing the installation position of a fisheye camera on a vehicle, and predefining a world coordinate system, a camera coordinate system, an imaging plane coordinate system and a pixel coordinate system, comprising:

[0006] According to an image captured by the fisheye camera, obtaining the coordinates of a target point in the pixel coordinate system and the pre-defined coordinates of a principal point of the fisheye camera in the pixel coordinate system, the target point being used to

[0007] representing a target recognized by the fisheye camera;

[0008] According to the coordinates of the target point in the pixel coordinate system and the pre-defined coordinates of the principal point of the fisheye camera in the pixel coordinate system, obtaining the imaging distortion coordinates of the target point in the imaging plane coordinate system;

[0009] According to the imaging distortion coordinates of the target point in the imaging plane coordinate system, determining the imaging non-distortion coordinates of the target point in the imaging plane coordinate system;

[0010] Converting the imaging non-distortion coordinates of the target point in the imaging plane coordinate system to the pixel coordinate system to obtain non-distortion target coordinates;

[0011] According to the target coordinates, the pre-defined internal parameters and external parameters of the fisheye camera, solving the coordinate values of the target point in the world coordinate system.

[0012] According to the coordinate value of the target point in the world coordinate system, a distance between the vehicle and the target point is determined.

[0013] In a possible implementation, the obtaining of the imaging distortion coordinate of the target point in the imaging plane coordinate system according to the coordinate of the target point in the pixel coordinate system and the pre-calibrated coordinate of the principal point of the fisheye camera in the pixel coordinate system comprises:

[0014] The imaging distortion coordinate of the target point in the imaging plane coordinate system is calculated according to a first preset formula, and the first preset formula is

[0015]

[0016] where (x', y') is used to represent the coordinate of the imaging distortion coordinate in the imaging plane coordinate system, (u', v') is used to represent the coordinate of the target point in the pixel coordinate system, (c x , c y ) is used to represent the coordinate of the principal point of the fisheye camera in the pixel coordinate system, f x and f y are used to represent the focal length of the fisheye camera, c x , c y , f x , f y are obtained from the internal parameters pre-calibrated by the fisheye camera.

[0017] In a possible implementation, the determining of the imaging non-distortion coordinate of the target point in the imaging plane coordinate system according to the imaging distortion coordinate of the target point in the imaging plane coordinate system comprises:

[0018] According to the imaging distortion coordinate, a distorted refraction angle of the target point when passing through the fisheye camera is calculated;

[0019] According to the refraction angle, an incident angle of the target point when passing through the fisheye camera is solved through a preset function, and the preset function is used to represent the functional relationship between the refraction angle and the incident angle;

[0020] According to the incident angle, the refraction angle and the imaging distortion coordinate, the imaging non-distortion coordinate is determined.

[0021] In a possible implementation, the calculating of the distorted refraction angle of the target point when passing through the fisheye camera according to the imaging distortion coordinate and the focal length of the fisheye camera comprises:

[0022] The refraction angle is calculated through a second preset formula, and the second preset formula is

[0023]

[0024] wherein θ d for representing the refraction angle, (x', y') for representing the coordinates of the imaging distortion coordinates in the imaging plane coordinate system,

[0025] the preset function is represented by a third preset formula, the third preset formula being

[0026] f(θ) = θ(1 + K1θ 2 + K2θ 4 + K3θ 6 + K4θ 8 )- θ d

[0027] wherein θ for representing the incident angle, f(θ) for representing a one-variable high-order function of θ, K1 for a first preset value, K2 for a second preset value, K3 for a third preset value, K4 for a fourth preset value, K1, K2, K3, K4 for distortion parameters obtained by pre-calibration of the fisheye camera.

[0028] In a possible implementation, the solving of the incident angle of the target point when passing through the fisheye camera according to the refraction angle by a preset function comprises:

[0029] setting the solving range as [low, up], initializing low = 0,

[0030] performing a judging step, the judging step comprising letting mid = (up + low) / 2, judging the relationship of f(mid) x f(low), if f(mid) x f(low) > 0, then letting low = mid, if f(mid) x f(low) < 0, then letting up = mid, wherein up for representing the maximum angle value in the current judging step, low for representing the minimum angle value in the current judging step, mid for representing the average value of the maximum angle value and the minimum angle value in the current judging step, f(mid) for representing the third preset formula with θ = mid, f(low) for representing the third preset formula with θ = low;

[0031] if a preset loop condition is met, repeating the judging step, if the loop condition is not met, terminating the judging step, and taking the mid obtained after the last execution of the judging step as the value of the incident angle, wherein the loop condition is that the absolute value of the difference between up and low after the completion of the current judging step is greater than or equal to a preset angle value.

[0032] In a possible implementation, the determining the imaging non-distorted coordinate according to the incident angle, the refraction angle and the imaging distortion coordinate comprises:

[0033] determining the imaging non-distorted coordinate according to a fourth preset formula, the fourth preset formula being

[0034]

[0035] wherein (x0, y0) is used to represent the imaging non-distorted coordinate in the imaging plane coordinate system.

[0036] In a possible implementation, the converting the imaging non-distorted coordinate of the target point in the imaging plane coordinate system into the pixel coordinate system to obtain the non-distorted target coordinate comprises:

[0037] determining the target coordinate according to a fifth preset formula, the fifth preset formula being

[0038]

[0039] wherein (u x , u y ) is used to represent the target coordinate in the pixel coordinate system, (x0, y0) is used to represent the imaging non-distorted coordinate in the imaging plane coordinate system, (c x , c y ) is used to represent the principal point of the fisheye camera in the pixel coordinate system, f x and f y are used to represent the focal length of the fisheye camera, c x , c y , f x , f y are obtained from the pre-calibrated internal parameters of the fisheye camera. In a possible implementation, the solving the coordinate value of the target point in the world coordinate system according to the target coordinate, the pre-calibrated internal parameters and external parameters of the fisheye camera comprises:

[0040] calculating the coordinate value of the target point in the world coordinate system according to a sixth preset formula, the sixth preset formula being

[0041]

[0042] wherein P w is used to represent the coordinate matrix of the target point in the world coordinate system, u x is used to represent the coordinate value of the target coordinate in the x-axis of the pixel coordinate system, u yR and T are used for representing the external parameter matrix of the fisheye camera pre-calibration, R -1 is used for representing the inverse matrix of R, s is used for representing a preset scale parameter, K is used for representing the internal parameter matrix of the fisheye camera, K -1 is used for representing the inverse matrix of K.

[0043] In a possible implementation, the method for determining the target point comprises:

[0044] Through a preset target recognition model, the image collected by the fisheye camera is recognized to obtain a recognition result, the recognition result being a target frame containing a target;

[0045] Any point on the lower bottom side of the target frame is taken as the target point.

[0046] In a second aspect, the present application provides a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to the possible implementation of the first aspect.

[0047] The present application embodiment obtains the coordinate value of the target point representing the recognized target in the distortion state of the imaging plane coordinate system, combines the refraction angle and the incidence angle of the fisheye camera imaging to obtain the corrected coordinate value of the camera in the non-distortion state, maps the coordinate value to the pixel coordinate system to obtain the target coordinate of the camera in the non-distortion state in the pixel coordinate system, combines the internal parameter and the external parameter of the fisheye camera to obtain the coordinate value of the target point in the world coordinate system representing the real world, and determines the distance between the vehicle and the target according to the coordinate value of the target point in the world coordinate system. Through the method provided by the present application embodiment, the ranging accuracy based on the fisheye camera is improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0049] Figure 1 is the implementation flowchart of the fisheye camera ranging method provided by the present application embodiment;

[0050] Figure 2 is the mapping relationship diagram of the world coordinate system, the camera coordinate system, the imaging plane coordinate system and the pixel coordinate system provided by the present application embodiment;

[0051] Figure 3 is a schematic diagram of a target point determination method provided by an embodiment of the present application;

[0052] Figure 4 is a mapping relationship diagram of a camera coordinate system and an imaging plane coordinate system of a fisheye camera provided by an embodiment of the present application;

[0053] Figure 5 is a structural schematic diagram of a fisheye camera ranging device provided by an embodiment of the present application;

[0054] Figure 6 is a schematic diagram of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the following description, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the present application. However, persons skilled in the art will understand that the present application can be practiced without these specific details. In other instances, well-known structures, devices, circuits, and methods have not been described in detail in order to avoid obscuring the present application.

[0056] In order to make the objectives, technical solutions and advantages of the present application clearer, the following will be described by specific embodiments in conjunction with the accompanying drawings.

[0057] Referring to Figure 1 which shows an implementation flowchart of a fisheye camera ranging method provided by an embodiment of the present application, and is described in detail as follows:

[0058] In step 101, according to an image captured by a fisheye camera, a coordinate of a target point in a pixel coordinate system and a pre-defined coordinate of a principal point of the fisheye camera in the pixel coordinate system are obtained, the target point being used to represent a target recognized by the fisheye camera.

[0059] In the embodiment of the present application, a world coordinate system, a camera coordinate system, an imaging plane coordinate system and a pixel coordinate system are defined in advance. Figure 2 is a mapping relationship diagram of a world coordinate system, a camera coordinate system, an imaging plane coordinate system and a pixel coordinate system provided by an embodiment of the present application, in conjunction with Figure 2 In the embodiment of the present application, four coordinate systems are defined in advance, which are a world coordinate system O w -X w Y w Z w , a camera coordinate system O c X c Y c Z c , an imaging plane coordinate system o′-xy and a pixel coordinate system o-uv.

[0060] The purpose of setting up a world coordinate system is to uniformly describe the position of objects in the real three-dimensional world, including the position of the camera, the position of the target, etc.

[0061] The camera coordinate system serves as a medium between three-dimensional spatial information and two-dimensional image information. For example... Figure 2 As shown, Z c The axis is the optical axis of the camera, Z. c The positive axis is outward along the camera's optical axis, X c O c Y c The plane is perpendicular to the camera's optical axis, X c The x-axis and Y-axis are parallel to the imaging plane coordinate system. c The y-axis is parallel to the imaging plane coordinate system.

[0062] In this embodiment of the invention, to simplify the calculation, the rule for setting the world coordinate system is as follows: obtain a perpendicular line passing through the camera's mounting point and perpendicular to the ground, and take the intersection of the perpendicular line and the ground as the origin O of the world coordinate system. w Using the perpendicular line as the Z-axis of the world coordinate system w Axis, Z w The positive direction of the axis is upward, and the X-axis of the world coordinate system is upward. w O w Y w plane and Z w The axis is perpendicular.

[0063] The imaging plane coordinate system is used to describe the position coordinates of feature points on the image plane, and the coordinate unit is generally millimeters. For example... Figure 2 As shown, the imaging plane coordinate system is a two-dimensional coordinate system established on the image plane, with the origin o′ being the Z-axis of the camera coordinate system. c The intersection of the x-axis and the image plane, with the x-axis and y-axis parallel to the X-axis in the camera coordinate system. c Axis, Y c Axis. The origin o′ of the imaging plane coordinate system and the origin O of the camera coordinate system. c The distance between them is the focal length f of the camera.

[0064] Images captured by a camera are typically two-dimensional digital images, stored as a pixel matrix, with each element representing the grayscale value of that pixel. The imaging plane coordinate system describes the position coordinates of an image point on the image plane, while the pixel coordinate system describes the point's position within the pixel matrix. For example... Figure 2 As shown, the pixel coordinate system is established on the image plane, with the origin o at the upper left corner of the image plane, the u-axis parallel to the x-axis of the imaging plane coordinate system, and the v-axis parallel to the y-axis of the imaging plane coordinate system.

[0065] For a point in the world coordinate system to its corresponding mapping point in the image plane, the pixel coordinates, the transformation between them includes three transformation processes, in turn: the transformation from the world coordinate system to the camera coordinate system, the transformation from the camera coordinate system to the imaging plane coordinate system, and the transformation from the imaging plane coordinate system to the pixel coordinate system.

[0066] Wherein, in combination Figure 2 , the camera coordinate system and the imaging plane coordinate system only have a proportional relationship, the focal length of the camera is the only influencing factor, and the transformation from the imaging plane coordinate system to the pixel coordinate system is only related to the physical size of the pixel in the x-axis direction of the imaging plane coordinate system and the physical size of the pixel in the y-axis direction of the imaging plane coordinate system. For a certain camera, the focal length f, the physical size of the pixel in the x-axis direction of the imaging plane coordinate system, and the physical size of the pixel in the y-axis direction of the imaging plane coordinate system are fixed.

[0067] The above four coordinate systems are only an example provided by the embodiment of the present application. Any four coordinate systems defined based on the concept of the embodiment of the present application are within the protection scope of the embodiment of the present application.

[0068] When the four coordinate systems are set, the mapping relationship between the four coordinate systems is uniquely determined.

[0069] After the four coordinate systems are constructed in advance, a target point used to represent the recognized target is determined in the image recognized by the fisheye camera. The coordinates of the target point in the pixel coordinate system are determined.

[0070] In an optional implementation, the method for determining the target point comprises: identifying the image collected by the fisheye camera through a pre-set target recognition model to obtain an identification result, the identification result being a target frame containing the target; and taking any point on the lower bottom side of the target frame as the target point.

[0071] Figure 3 is a schematic diagram of a target point determination method provided by the embodiment of the present application, in combination Figure 3 , the original image of the fisheye camera is obtained by acquiring the image collected by the fisheye camera in real time. Optionally, to improve the processing efficiency and reduce the calculation amount, the original image can be scaled, such as being scaled to 768x384, to improve the running efficiency. The scaled image is input into a pre-set target recognition model, such as a neural network model, the target recognition model is used to identify the target in the image, and a target frame of the target is obtained, as shown in Figure 3 .

[0072] Any point on the lower bottom side line segment AB of the target frame is taken as the target point, and preferably, the center point C of the lower bottom side line segment AB is taken as the target point.

[0073] In the embodiment of the present application, the coordinates of the target point in the pixel coordinate system are denoted as (x1, y1), and the coordinates of the principal point of the fisheye camera pre-calibrated in the pixel coordinate system are denoted as (c x , c y ). x , c y ) are determined by the internal parameters pre-calibrated by the fisheye camera.

[0074] In step 102, the imaging distortion coordinates of the target point in the imaging plane coordinate system are obtained according to the coordinates of the target point in the pixel coordinate system and the coordinates of the principal point of the fisheye camera pre-calibrated in the pixel coordinate system.

[0075] wherein the imaging distortion coordinates are the coordinates of the target point in the imaging plane coordinate system in the fisheye camera in the distortion state.

[0076] In an alternative implementation, the difference between the coordinate value of the target point in the x-axis of the pixel coordinate system and the coordinate value of the principal point of the fisheye camera in the x-axis of the pixel coordinate system is divided by the focal length of the fisheye camera in the x-axis to obtain the coordinate value of the imaging distortion coordinates in the x-axis of the imaging plane coordinate system; and the difference between the coordinate value of the target point in the y-axis of the pixel coordinate system and the coordinate value of the principal point of the fisheye camera in the y-axis of the pixel coordinate system is divided by the focal length of the fisheye camera in the y-axis to obtain the coordinate value of the imaging distortion coordinates in the y-axis of the imaging plane coordinate system.

[0077] In an alternative implementation, the imaging distortion coordinates of the target point in the imaging plane coordinate system are calculated according to a first preset formula, which is

[0078]

[0079] wherein (x', y') is used to represent the coordinates of the imaging distortion coordinates in the imaging plane coordinate system, (u', v') is used to represent the coordinates of the target point in the pixel coordinate system, (c x , c y ) is used to represent the coordinates of the principal point of the fisheye camera in the pixel coordinate system, f x and f y are used to represent the focal length of the fisheye camera, and c x , c y , f x , f y are obtained from the internal parameters pre-calibrated by the fisheye camera.

[0080] wherein the principal point of the fisheye camera is the intersection of the principal optical axis of the fisheye camera and the image plane.

[0081] In step 103, the imaging non-distortion coordinates of the target point in the imaging plane coordinate system are determined according to the imaging distortion coordinates of the target point in the imaging plane coordinate system.

[0082] The imaging non-distortion coordinates are the coordinates of the target point in the imaging plane coordinate system in a state where the fisheye camera does not have distortion.

[0083] In an optional implementation, the fisheye camera is corrected for distortion to obtain the coordinates of the corrected target point in the imaging plane coordinate system, that is, the coordinates of the target point in the imaging plane coordinate system in a state where the fisheye camera does not have distortion.

[0084] The embodiment of the application does not limit the method for correcting the distortion of the fisheye camera.

[0085] In an optional implementation, the distortion angle of the target point passing through the fisheye camera is calculated according to the imaging distortion coordinates; the incident angle of the target point passing through the fisheye camera is solved by a preset function according to the distortion angle, the preset function being used to represent the functional relationship between the distortion angle and the incident angle; and the imaging non-distortion coordinates are determined according to the incident angle, the distortion angle, and the imaging distortion coordinates.

[0086] Figure 4 is a mapping relationship diagram of the camera coordinate system and the imaging plane coordinate system of the fisheye camera provided by the embodiment of the application. For example, the target point corresponds to a point P in the camera coordinate system O c -X c Y c Z c , θ is the incident angle of the target point P, θ d is the equivalent distortion angle, P'(x', y') is the coordinate after distortion, corresponding to the imaging distortion coordinates in step 102. P0(x0, y0) is the coordinate without distortion, corresponding to the imaging non-distortion coordinates in the present step.

[0087] In combination with Figure 4 , O'P' = r d , O'P0 = r.

[0088] Since the distortion of the fisheye camera is irrelevant to the focal length, f = 1 is taken, and in combination with the geometric relationship,

[0089] Considering that the imaging CCD (charge coupled device) plane size of the fisheye camera is generally several millimeters, and the focal length is about several hundred millimeters, θ d is relatively small in the actual imaging process of the camera, so tanθ d ≈ θ d .

[0090] Based on this, in an optional implementation, the refraction angle is calculated by a second preset formula, the second preset formula is

[0091]

[0092] wherein, θ d is used to represent the refraction angle, and x' is used to represent the x-axis coordinate in the imaging distortion coordinate, and y' is used to represent the y-axis coordinate in the imaging distortion coordinate.

[0093] The incident angle θ is generally an acute angle, so the definition range is According to the equirectangular projection formula and the Taylor expansion, a third preset formula is approximately obtained, the preset function is represented by the third preset formula, and the third preset formula is

[0094] f(θ) = θ (1 + K1θ 2 + K2θ 4 + K3θ 6 + K4θ 8 )- θ d

[0095] wherein, θ is used to represent the incident angle, f(θ) is used to represent the one-variable high-order function of θ, K1 is a first preset value, K2 is a second preset value, K3 is a third preset value, K4 is a fourth preset value, and K1, K2, K3 and K4 are distortion parameters obtained by the fish-eye camera pre-calibration.

[0096] In an optional implementation, the solution range is set as [low, up], low = 0 is initialized, A judgment step is performed, the judgment step includes mid = (up + low) / 2, the relationship of f(mid) x f(low) is judged, if f(mid) x f(low) > 0, low = mid is set, if f(mid) x f(low) < 0, up = mid is set, wherein, up is used to represent the maximum angle value in the execution of the current judgment step, low is used to represent the minimum angle value in the execution of the current judgment step, mid is used to represent the average value of the maximum angle value and the minimum angle value in the execution of the current judgment step, f(mid) is used to represent θ = mid substituted into the third preset formula, and f(low) is used to represent θ = low substituted into the third preset formula; if a preset loop condition is met, the judgment step is repeatedly executed, if the loop condition is not met, the judgment step is terminated, and the mid obtained after the last execution of the judgment step is taken as the value of the incident angle, wherein, the loop condition is that the absolute value of the difference between up and low after the completion of the current judgment step is greater than or equal to a preset angle value.

[0097] The preset angle range is The initial value of low and the initial value of up are 0 and

[0098] In one specific example, the preset angle value in the loop condition is 0.01°, and the loop condition is |up-low|≥0.01°.

[0099] The first execution judgment step is performed, and low=0, The relationship of f(mid)xf(low) is judged. If f(mid)xf(low)>0, then If f(mid)xf(low)<0, then

[0100] If f(mid)xf(low)>0, then For example, after the first judgment step is executed, The loop condition is satisfied, and the judgment step is executed again until the loop condition is not satisfied. The mid obtained after the last execution of the judgment step is taken as the value of the incident angle θ.

[0101] According to the third preset formula, the distortion of the fisheye camera is irrelevant to the focal length f of the camera, so f can be any value. In an optional implementation manner, f=1. At this time, combined with Figure 4 , d d .

[0102] Combined with Figure 4 According to similar triangles, the following can be obtained:

[0103]

[0104]

[0105] Based on this, in an optional implementation manner, the imaging non-distorted coordinate is determined according to the fourth preset formula. The fourth preset formula is

[0106]

[0107] Where (x0, y0) is used to represent the coordinates of the imaging non-distorted coordinate in the imaging plane coordinate system.

[0108] In step 104, the imaging non-distorted coordinate of the target point in the imaging plane coordinate system is converted into the pixel coordinate system to obtain the non-distorted target coordinate.

[0109] When the imaging non-distorted coordinate P0(x0, y0) is determined, the imaging non-distorted coordinate is converted into the pixel coordinate system according to the unique mapping relationship between the imaging plane coordinate system and the pixel coordinate system to obtain the target coordinate in the pixel coordinate system.

[0110] ​In an alternative implementation, the target coordinates are determined according to a fifth preset formula

[0111]

[0112] wherein (u x , u y ) represent the target coordinates in the pixel coordinate system, (x0, y0) represent the imaging non-distorted coordinates in the imaging plane coordinate system, (c x , c y ) represent the principal point of the fisheye camera in the pixel coordinate system, f x and f y represent the focal length of the fisheye camera, c x , c y , f x , f y are obtained from the pre-calibrated internal parameters of the fisheye camera.

[0113] In step 105, the coordinates of the target point in the world coordinate system are solved according to the target coordinates, the pre-calibrated internal parameters and the external parameters of the fisheye camera.

[0114] wherein the external parameters represent the conversion relationship between the world coordinate system and the camera coordinate system, and the internal parameters include the pre-calibrated internal parameters, including c x , c y , f x and f y .

[0115] Since the four coordinate systems are defined, the mapping relationship between the four coordinate systems is uniquely determined. A point in the pixel coordinate system can be uniquely mapped to a point in the world coordinate system.

[0116] In an alternative implementation, the coordinates of the target point in the world coordinate system are calculated according to a sixth preset formula

[0117]

[0118] wherein P w represents the coordinate matrix of the target point in the world coordinate system, u x represents the coordinate value of the target coordinates in the x-axis of the pixel coordinate system, u y represents the coordinate value of the target coordinates in the y-axis of the pixel coordinate system, R and T represent the pre-calibrated external parameter matrix of the fisheye camera, R -1 represents the inverse matrix of R, s represents a preset scale parameter, K represents the internal parameter matrix of the fisheye camera, and K -1 represents the inverse matrix of K.

[0119] wherein s is the value of the target point in the z-axis direction of the camera coordinate system.

[0120] In step 106, the distance between the vehicle and the target point is determined according to the coordinate value of the target point in the world coordinate system.

[0121] In step 105, P w The coordinate matrix of the target point in the world coordinate system is used to represent the coordinate value of the target point in the three axes of the world coordinate system. Since the world coordinate system is used to uniformly describe the position of an object in a real three-dimensional world, the coordinate of the target point in the world coordinate system represents the position of the target point in the real world.

[0122] In an optional implementation, the absolute value of the x-axis value of a point in the world coordinate system is used to represent the lateral distance from the vehicle, and the lateral distance is used to represent the vertical distance of the target from the side of the vehicle body. For example, the fisheye camera in the embodiment of the present application is applied to the BSD (Blind-Spot-Detection, blind spot detection system) of the vehicle and is installed on the side of the vehicle. Therefore, the absolute value of the x-axis value of the target in the world coordinate system is the distance from the side of the vehicle body. In the embodiment of the present application, the absolute value of the y-axis value of a point in the world coordinate system is used to represent the longitudinal distance from the vehicle, and the longitudinal distance is used to represent the vertical distance of the target from the front of the vehicle or the parking space. For example, the fisheye camera in the embodiment of the present application is installed at the front position of the vehicle. Therefore, the absolute value of the y-axis value of the target in the world coordinate system is the distance from the front of the vehicle.

[0123] In another optional implementation, the absolute value of the x-axis value of a point in the world coordinate system is used to represent the longitudinal distance from the vehicle, and the absolute value of the y-axis value of a point in the world coordinate system is used to represent the lateral distance from the vehicle.

[0124] The embodiment of the present application does not limit this.

[0125] In the embodiment of the present application, if the fisheye camera is installed on the side of the vehicle, the lateral distance is the target distance, and if the analyzed target distance is less than the preset braking distance, the braking is triggered. If the fisheye camera is installed at the front position of the vehicle, the longitudinal distance is the target distance, and if the analyzed target distance is less than the preset braking distance, the braking is triggered. Optionally, the preset braking distances corresponding to the lateral distance and the longitudinal distance can be the same or different, which is set according to the actual application scene, and the embodiment of the present application does not limit this.

[0126] The embodiment of the present application obtains the coordinate value of a target point representing a target in a distorted state in an imaging plane coordinate system, obtains the coordinate value of the camera in a non-distorted state after correction in combination with the refraction angle and the incidence angle of the fisheye camera imaging, maps the target coordinate in the pixel coordinate system in the non-distorted state of the camera to obtain the target coordinate in the pixel coordinate system in the non-distorted state of the camera, obtains the coordinate value of the target point in a world coordinate system representing the real world in combination with the internal parameters and the external parameters of the fisheye camera, and determines the distance between the vehicle and the target according to the coordinate value of the target point in the world coordinate system. The method provided by the embodiment of the present application improves the ranging accuracy based on the fisheye camera.

[0127] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0128] The following is the device embodiment of the present application. For details not described in detail, please refer to the corresponding method embodiments described above.

[0129] Figure 5 The structure schematic diagram of the fisheye camera ranging device provided by the embodiment of the present application is shown. For the convenience of description, only the part related to the embodiment of the present application is shown, and the details are as follows:

[0130] As shown in Figure 5 , the fisheye camera ranging device 5 comprises a first determining module 51, a second determining module 52, a third determining module 53, a fourth determining module 54, an analysis module 55 and a distance determining module 56;

[0131] The first determining module 51 is configured to obtain the coordinate of the target point in the pixel coordinate system and the pre-designated coordinate of the principal point of the fisheye camera in the pixel coordinate system according to the image captured by the fisheye camera, and the target point is used to represent the target recognized by the fisheye camera;

[0132] The second determining module 52 is configured to obtain the imaging distortion coordinate of the target point in the imaging plane coordinate system according to the coordinate of the target point in the pixel coordinate system and the pre-designated coordinate of the principal point of the fisheye camera in the pixel coordinate system;

[0133] The third determining module 53 is configured to determine the imaging non-distortion coordinate of the target point in the imaging plane coordinate system according to the imaging distortion coordinate of the target point in the imaging plane coordinate system;

[0134] The fourth determining module 54 is configured to convert the imaging non-distortion coordinate of the target point in the imaging plane coordinate system to the pixel coordinate system to obtain the non-distortion target coordinate;

[0135] The solving module 55 is configured to solve the coordinate value of the target point in the world coordinate system according to the target coordinate, the internal parameter and the external parameter of the fisheye camera pre-calibration.

[0136] The distance determining module 56 is configured to determine the distance between the vehicle and the target point according to the coordinate value of the target point in the world coordinate system.

[0137] The embodiment of the present application obtains the coordinate value of the target point in the distortion state in the imaging plane coordinate system, obtains the corrected coordinate value of the camera in the non-distortion state by combining the incident angle and the refraction angle of the fisheye camera imaging, maps the target coordinate in the pixel coordinate system in the non-distortion state of the camera, combines the internal parameter and the external parameter of the fisheye camera, obtains the coordinate value of the target point in the world coordinate system representing the real world, and determines the distance between the vehicle and the target according to the coordinate value of the target point in the world coordinate system. The method provided by the embodiment of the present application improves the ranging accuracy based on the fisheye camera.

[0138] In a possible implementation, the second determining module 52 is configured to calculate the imaging distortion coordinate of the target point in the imaging plane coordinate system according to a first preset formula, the first preset formula being

[0139]

[0140] wherein (x', y') is used to represent the coordinate of the imaging distortion coordinate in the imaging plane coordinate system, (u', v') is used to represent the coordinate of the target point in the pixel coordinate system, (c x , c y ) is used to represent the coordinate of the principal point of the fisheye camera in the pixel coordinate system, f x and f y are used to represent the focal length of the fisheye camera, c x , c y , f x , f y are obtained from the internal parameter pre-calibrated by the fisheye camera.

[0141] In a possible implementation, the third determining module 53 is configured to calculate the distortion refraction angle of the target point when passing through the fisheye camera according to the imaging distortion coordinate and the focal length of the fisheye camera.

[0142] According to the refraction angle, the incident angle of the target point when passing through the fisheye camera is solved by a preset function, the preset function being used to represent the functional relationship between the refraction angle and the incident angle.

[0143] The imaging non-distortion coordinate is determined according to the incident angle, the refraction angle and the imaging distortion coordinate.

[0144] In a possible implementation, the third determining module 53 is further configured to calculate the refraction angle by a second preset formula, the second preset formula being

[0145]

[0146] wherein, θ d is used to represent the refraction angle, (x', y') is used to represent the coordinates of the imaging distortion coordinates in the imaging plane coordinate system, and f is used to represent the preset function.

[0147] The preset function is represented by a third preset formula, the third preset formula being

[0148] f(θ) = θ(1 + K1θ 2 + K2θ 4 + K3θ 6 + K4θ 8 )- θ d

[0149] wherein, θ is used to represent the incident angle, f(θ) is used to represent a one-variable high-order function of θ, K1 is a first preset value, K2 is a second preset value, K3 is a third preset value, and K4 is a fourth preset value, and K1, K2, K3, and K4 are distortion parameters obtained by pre-calibrating the fisheye camera.

[0150] In a possible implementation, the third determining module 53 is further configured to:

[0151] Set the solution range as [low, up], and initialize low = 0, perform a judgment step, the judgment step including setting mid = (up + low) / 2, judging the relationship between f(mid) and f(low), setting low = mid if f(mid) * f(low) > 0, setting up = mid if f(mid) * f(low) < 0, wherein up is used to represent the maximum value of the angle in the current judgment step, low is used to represent the minimum value of the angle in the current judgment step, mid is used to represent the average value of the maximum and minimum values of the angle in the current judgment step, f(mid) is used to represent the third preset formula with θ = mid, and f(low) is used to represent the third preset formula with θ = low;

[0152] if a preset loop condition is met, repeatedly performing the judgment step, if the loop condition is not met, terminating the judgment step, and taking mid obtained after the last execution of the judgment step as the value of the incident angle, wherein the loop condition is that the absolute value of the difference between up and low after the current judgment step is greater than or equal to a preset angle value.

[0153] In a possible implementation, the third determining module 53 is further configured to:

[0154] The imaging non-distorted coordinate is determined according to a fourth preset formula, the fourth preset formula being

[0155]

[0156] Wherein, (x0, y0) is used to represent the coordinate of the imaging non-distorted coordinate in the imaging plane coordinate system.

[0157] In a possible implementation, the fourth determination module 54 is further configured to:

[0158] The target coordinate is determined according to a fifth preset formula, the fifth preset formula being

[0159]

[0160] Wherein, (u x , u y ) is used to represent the coordinate of the target coordinate in the pixel coordinate system, (x0, y0) is used to represent the coordinate of the imaging non-distorted coordinate in the imaging plane coordinate system, (c x , c y ) is used to represent the principal point of the fisheye camera in the pixel coordinate system, f x and f y are used to represent the focal length of the fisheye camera, c x , c y , f x , f y are obtained from the pre-calibration internal parameters of the fisheye camera. In a possible implementation, the analysis module 55 is configured to:

[0161] The coordinate value of the target point in the world coordinate system is calculated according to a sixth preset formula, the sixth preset formula being

[0162]

[0163] Wherein, P w is used to represent the coordinate matrix of the target point in the world coordinate system, u x is used to represent the coordinate value of the target coordinate in the x-axis of the pixel coordinate system, u y is used to represent the coordinate value of the target coordinate in the y-axis of the pixel coordinate system, R and T are used to represent the pre-calibration external parameter matrix of the fisheye camera, R -1 is used to represent the inverse matrix of R, s is used to represent a preset scale parameter, K is used to represent the internal parameter matrix of the fisheye camera, K -1 is used to represent the inverse matrix of K.

[0164] In a possible implementation, the first determination module 51 is configured to:

[0165] The image collected by the fisheye camera is identified by a pre-set target recognition model to obtain an identification result, and the identification result is a target frame containing the target.

[0166] Any point on the lower bottom edge of the target frame is taken as the target point.

[0167] The embodiments of the present application also provide a computer program product having program codes which, when running in a corresponding processor, controller, computing device or terminal, perform the steps in any one of the above-mentioned fisheye camera ranging method embodiments, such as steps fisheye camera ranging method to fisheye camera ranging method shown in FIG. X. Those skilled in the art should understand that the method and the device proposed by the embodiments of the present application can be realized in various forms of hardware, software, firmware, special-purpose processor or combination thereof. The special-purpose processor can include application-specific integrated circuit (ASIC), reduced instruction set computer (RISC) and / or field programmable gate array (FPGA). The proposed method and device are preferably realized as a combination of hardware and software. The software is preferably installed as an application program on a program storage device. It is typically a machine based on a computer platform with hardware, such as one or more central processing units (CPUs), random access memories (RAMs) and one or more input / output (I / O) interfaces. An operating system is also typically installed on the computer platform. The various processes and functions described herein can be part of the application program, or part thereof can be executed by the operating system.

[0168] Figure 6 is a schematic diagram of a terminal provided by the embodiments of the present application. As shown in Figure 6 , the terminal 6 of the embodiments includes a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. The processor 60 implements the steps in the above-mentioned various fisheye camera ranging method embodiments when executing the computer program 62, such as Figure 1 steps 101 to 106 shown in FIG. X. Alternatively, the processor 60 implements the functions of the modules / units in the above-mentioned various device embodiments when executing the computer program 62, such as Figure 5 the functions of the modules / units 51 to 56 shown in FIG. X.

[0169] For example, the computer program 62 can be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete / implement the schemes provided by the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 62 in the terminal 6. For example, the computer program 62 can be divided intoFigure 5 The modules / units 51 to 56 are shown.

[0170] The terminal 6 can be a vehicle terminal. The terminal 6 can include, but is not limited to, a processor 60, a memory 61. Those skilled in the art can understand that the terminal 6 can include more or less components than those shown, or combine some components, or different components, for example, the terminal can also include an input / output device, a network access device, a bus, etc. Figure 6 The terminal 6 shown is merely an example and does not constitute a limitation on the terminal 6, and can include more or less components than those shown, or combine some components, or different components, for example, the terminal can also include an input / output device, a network access device, a bus, etc.

[0171] The processor 60 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0172] The memory 61 can be an internal storage unit of the terminal 6, such as a hard disk or a memory of the terminal 6. The memory 61 can also be an external storage device of the terminal 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 61 can include both an internal storage unit and an external storage device of the terminal 6. The memory 61 is used to store the computer program and other programs and data required by the terminal. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0173] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific name of each functional unit and module is only for the convenience of mutual distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

[0174] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0175] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0176] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutually can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0177] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0178] In addition, each of the function units in each of the embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0179] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of each fisheye camera ranging method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0180] In addition, the features of the embodiments shown in the drawings of the present application or mentioned in the specification of the present application do not have to be understood as independent embodiments from each other. Rather, each feature described in one example of an embodiment can be combined with one or more other desired features from other embodiments, thereby resulting in other embodiments which have not been described in words or with reference to the drawings.

[0181] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A fisheye camera ranging method, characterized in that, The fisheye camera is mounted in a fixed position on the vehicle, and the world coordinate system, camera coordinate system, imaging plane coordinate system, and pixel coordinate system are predefined, including: Based on the image captured by the fisheye camera, the coordinates of the target point in the pixel coordinate system and the pre-calibrated coordinates of the principal point of the fisheye camera in the pixel coordinate system are obtained. The target point is used to represent the target identified by the fisheye camera. Based on the coordinates of the target point in the pixel coordinate system and the pre-calibrated coordinates of the principal point of the fisheye camera in the pixel coordinate system, the imaging distortion coordinates of the target point in the imaging plane coordinate system are obtained. The undistorted coordinates of the target point in the imaging plane coordinate system are determined based on the distorted coordinates of the target point in the imaging plane coordinate system. The undistorted coordinates of the target point in the imaging plane coordinate system are transformed to the pixel coordinate system to obtain the undistorted target coordinates; Based on the target coordinates, the pre-calibrated internal and external parameters of the fisheye camera, the coordinates of the target point in the world coordinate system are calculated. The distance between the vehicle and the target point is determined based on the coordinates of the target point in the world coordinate system. Determining the undistorted coordinates of the target point in the imaging plane coordinate system based on the distortion coordinates of the target point in the imaging plane coordinate system includes: Based on the imaging distortion coordinates, calculate the distorted refraction angle of the target point when it passes through the fisheye camera; Based on the refraction angle, the incident angle when the target point passes through the fisheye camera is calculated using a preset function, whereby the preset function represents the functional relationship between the refraction angle and the incident angle. The undistorted coordinates of the image are determined based on the incident angle, the refraction angle, and the image distortion coordinates. The step of solving for the coordinates of the target point in the world coordinate system based on the target coordinates and the pre-calibrated internal and external parameters of the fisheye camera includes: The coordinates of the target point in the world coordinate system are calculated according to the sixth preset formula, which is: in, A matrix used to represent the coordinates of the target point in the world coordinate system. Used to represent the target coordinates in the pixel coordinate system The coordinate values ​​of the axis. Used to represent the target coordinates in the pixel coordinate system The coordinate values ​​of the axis. and Used to represent the pre-calibrated extrinsic parameter matrix of the fisheye camera. Used to represent The inverse matrix, Used to represent preset ratio parameters. This is used to represent the internal parameter matrix of the fisheye camera. Used to represent The inverse matrix; Determining the distance between the vehicle and the target point based on the coordinates of the target point in the world coordinate system includes: If the fisheye camera is mounted on the side of the vehicle, the distance between the vehicle and the target point is determined based on the lateral distance of the target point in the world coordinate system. If a fisheye camera is installed at the front of the vehicle, the distance between the vehicle and the target point is determined based on the longitudinal distance of the target point's coordinates in the world coordinate system.

2. The method according to claim 1, characterized in that, The step of obtaining the imaging distortion coordinates of the target point in the imaging plane coordinate system based on the coordinates of the target point in the pixel coordinate system and the pre-calibrated coordinates of the principal point of the fisheye camera in the pixel coordinate system includes: The imaging distortion coordinates of the target point in the imaging plane coordinate system are calculated according to a first preset formula, wherein the first preset formula is: in, Used to represent the coordinates of the imaging distortion coordinates in the imaging plane coordinate system. Used to represent the coordinates of the target point in the pixel coordinate system. This is used to represent the coordinates of the principal point of the fisheye camera in the pixel coordinate system. and Used to indicate the focal length of the fisheye camera. , , , It is obtained from the pre-calibrated internal parameters of the fisheye camera.

3. The method according to claim 1, characterized in that, The step of calculating the distorted refraction angle of the target point when it passes through the fisheye camera based on the imaging distortion coordinates includes: The refraction angle is calculated using a second preset formula, which is: in, Used to represent the angle of refraction. Used to represent the coordinates of the imaging distortion coordinates in the imaging plane coordinate system; The preset function is represented by a third preset formula, which is: in, Used to represent the angle of incidence, Used to represent A higher-order function of one variable The first preset value For the second preset value, The third preset value This is the fourth preset value. , , , The distortion parameters are obtained from the pre-calibration of the fisheye camera.

4. The method according to claim 3, characterized in that, The step of calculating the incident angle of the target point when it passes through the fisheye camera using a preset function based on the refraction angle includes: Set the solution range to [low, up], initialize low = 0, up = ; The judgment step includes setting mid = (up + low) / 2, judging the relationship between f(mid) × f(low), if f(mid) × f(low) > 0, then setting low = mid; if f(mid) × f(low) < 0, then setting up = mid, where up represents the maximum angle value at the time of this judgment step, low represents the minimum angle value at the time of this judgment step, mid represents the average of the maximum and minimum angle values ​​at the time of this judgment step, and f(mid) represents the value of the maximum and minimum angle values ​​at the time of this judgment step. Substituting into the third preset formula, f(low) is used to represent... Substitute into the third preset formula; If the preset loop condition is met, the judgment step is repeated. If the loop condition is not met, the judgment step is terminated, and the mid obtained after the last execution of the judgment step is taken as the value of the incident angle. The loop condition is that the absolute value of the difference between up and low after the completion of this judgment step is greater than or equal to the preset angle value.

5. The method according to claim 3, characterized in that, Determining the undistorted coordinates of the image based on the incident angle, the refraction angle, and the image distortion coordinates includes: The undistorted coordinates of the image are determined according to a fourth preset formula, which is: in, This is used to represent the coordinates of the undistorted imaging coordinates in the imaging plane coordinate system.

6. The method according to claim 1, characterized in that, The step of transforming the undistorted imaging coordinates of the target point in the imaging plane coordinate system to the pixel coordinate system to obtain the undistorted target coordinates includes: The target coordinates are determined according to a fifth preset formula, which is: in, Used to represent the coordinates of the target in the pixel coordinate system. Used to represent the coordinates of the undistorted imaging coordinates in the imaging plane coordinate system. This is used to represent the coordinates of the principal point of the fisheye camera in the pixel coordinate system. and Used to indicate the focal length of the fisheye camera. , , , It is obtained from the pre-calibrated internal parameters of the fisheye camera.

7. The method according to any one of claims 1 to 6, characterized in that, The method for determining the target point includes: The image captured by the fisheye camera is identified by a pre-set target recognition model to obtain a recognition result, which is a target bounding box containing the target. Take any point on the bottom edge of the target box as the target point.

8. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the fisheye camera ranging method as described in any one of claims 1 to 6.

Images