Target detection method and device based on fisheye camera and vehicle

Abstract

The application discloses a target detection method based on a fisheye camera, a device thereof and a vehicle. The target detection method comprises the following steps: acquiring a first conversion relationship between a camera coordinate system of the fisheye camera and a world coordinate system, and a second conversion relationship between the camera coordinate system of the fisheye camera and a pixel coordinate system; and determining corresponding coordinates of a target point in the world coordinate system in an image captured by the fisheye camera according to the first conversion relationship, the second conversion relationship and a deflection angle of an optical axis of the fisheye camera relative to a set plane in the world coordinate system. The optical center of the fisheye camera and the center of the target detection object do not need to be located on the same horizontal plane, different installation angles of the fisheye camera are supported, installation requirements and installation difficulty are reduced, and the use scene is more.

Description

Technical Field

[0001] This application relates to the field of cameras, and more specifically to a target detection method and apparatus based on a fisheye camera, and a vehicle thereof. Background Technology

[0002] Fisheye cameras, with their wide field of view, are used in various applications, including but not limited to vehicles and surveillance, for tasks such as target ranging. However, the installation of fisheye cameras requires ensuring that the optical center of the camera and the center of the target object are on the same horizontal plane, making installation more demanding and increasing the difficulty of installation. Summary of the Invention

[0003] This application is made to address at least one of the aforementioned problems. According to a first aspect of this application, a target detection method based on a fisheye camera is provided. The target detection method includes: obtaining a first transformation relationship between the camera coordinate system and the world coordinate system of the fisheye camera, and a second transformation relationship between the camera coordinate system and the pixel coordinate system of the fisheye camera; determining the corresponding coordinates of a target point in an image captured by the fisheye camera in the world coordinate system based on the first transformation relationship, the second transformation relationship, and the deflection angle of the optical axis of the fisheye camera relative to a set plane in the world coordinate system.

[0004] In one embodiment of this application, the target detection method further includes: the deflection angle is obtained in advance based on the installation position of the fisheye camera; or, the deflection angle is calculated based on the first conversion relationship.

[0005] In one embodiment of this application, the deflection angle between the optical axis of the fisheye camera and the set plane is not less than 0° and not greater than 90°.

[0006] In one embodiment of this application, determining the corresponding coordinates of a target point in the world coordinate system in an image captured by the fisheye camera based on the first transformation relationship, the second transformation relationship, and the deflection angle of the optical axis of the fisheye camera relative to a set plane in the world coordinate system includes: obtaining the first coordinates of the target point in the pixel coordinate system based on the image captured by the fisheye camera; determining the second coordinates of the target point in the camera coordinate system based on the first coordinates, the second transformation relationship, and the deflection angle; and determining the third coordinates of the target point in the world coordinate system based on the second coordinates and the first transformation relationship.

[0007] In one embodiment of this application, determining the second coordinates of the target point in the camera coordinate system based on the first coordinates, the second transformation relationship, and the deflection angle includes: obtaining the vertical distance of the target point from the set plane in the world coordinate system; and determining the second coordinates of the target point in the camera coordinate system based on the first coordinates, the second transformation relationship, the deflection angle, and the vertical distance.

[0008] In one embodiment of this application, the target detection method further includes: using a target detection algorithm to detect target objects on the image, and determining the target point based on the target objects.

[0009] In one embodiment of this application, the step of detecting a target object in the image using a target detection algorithm and determining the target point based on the target object includes: using a detection box of the target detection algorithm to frame the target object in the image; and determining the target point of the target object based on the detection box.

[0010] In one embodiment of this application, determining the target point of the target object based on the detection frame includes: taking the midpoint of the lower edge of the detection frame as the target point.

[0011] In one embodiment of this application, the target detection method further includes: determining the distance between the target point and the set point based on the third coordinate and the set coordinate of the set point in the world coordinate system.

[0012] In one embodiment of this application, determining the distance between the target point and the set point based on the third coordinate and the set point's set coordinates in the world coordinate system includes: determining a first projection point and a second projection point on the set plane, respectively, based on the third coordinate and the set coordinate, and determining the distance between the first projection point and the second projection point.

[0013] In one embodiment of this application, the set point is the origin of the world coordinate system or the optical center of the fisheye camera.

[0014] In one embodiment of this application, the set plane is parallel to both the first and second coordinate axes in the world coordinate system.

[0015] In one embodiment of this application, determining the first transformation relationship between the camera coordinate system and the world coordinate system of the fisheye camera includes: obtaining the coordinates of at least four calibration point pairs, each calibration point pair including: a pixel coordinate system calibration point and a world coordinate system calibration point corresponding to the pixel coordinate system calibration point; wherein, the coordinates of the pixel coordinate system calibration point are the coordinates of the pixel coordinate system calibration point in the pixel coordinate system, the coordinates of the world coordinate system calibration point are the coordinates of the world coordinate system calibration point in the world coordinate system, and all the world coordinate system calibration points are located in the set plane; determining the rotation vector and translation vector between the camera coordinate system and the world coordinate system based on the coordinates of the at least four calibration point pairs and the second transformation relationship; obtaining the extrinsic parameter matrix of the fisheye camera based on the rotation vector and the translation vector, and using the extrinsic parameter matrix as the first transformation relationship.

[0016] In one embodiment of this application, the second transformation relationship includes the intrinsic parameter matrix and distortion coefficients of the fisheye camera.

[0017] According to a second aspect of this application, a target detection device based on a fisheye camera is also provided. The target detection device includes a storage medium and a processor. The storage medium stores a computer program that is executed by the processor. When the computer program is executed by the processor, the processor causes the processor to perform any of the above-described target detection methods based on a fisheye camera.

[0018] According to a third aspect of this application, a vehicle is also provided, the vehicle comprising: a vehicle body, a fisheye camera mounted on the vehicle body, and any of the above-described target detection devices based on the fisheye camera.

[0019] According to the target detection method, apparatus, and vehicle using a fisheye camera provided in this application, the method first obtains a first transformation relationship between the camera coordinate system and the world coordinate system of the fisheye camera, and a second transformation relationship between the camera coordinate system and the pixel coordinate system of the fisheye camera. Then, based on the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system, the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system are determined. Compared to the prior art, which requires ensuring that the optical center of the fisheye camera and the center of the target object are on the same horizontal plane, this application, after completing the calibration of internal and external parameters, determines the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system based on the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system. This eliminates the need for the optical center of the fisheye camera to be on the same horizontal plane as the center of the target object, supports different installation angles for the fisheye camera, reduces installation requirements and difficulty, and expands the application scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a target detection method using a fisheye camera according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart illustrating a target detection method using a fisheye camera according to another embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram illustrating the imaging principle of a fisheye camera according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of a fisheye camera imaging in the YOZ plane in the world coordinate system, as shown in an embodiment of the present invention.

[0025] Figure 5 This is a diagram illustrating the positional relationships between various coordinate systems according to an embodiment of the present invention;

[0026] Figure 6 This is a schematic block diagram of a target detection device using a fisheye camera according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0028] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0029] It should be understood that the invention can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “compose” and / or “comprising,” when used in this specification, confirm the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0031] To fully understand this invention, a detailed structure will be presented in the following description to illustrate the technical solution proposed by this invention. Optional embodiments of the invention are described in detail below; however, in addition to these detailed descriptions, the invention may have other embodiments.

[0032] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0033] First, let me introduce the application scenarios of the target detection method based on a fisheye camera illustrated in this application. This target detection method based on a fisheye camera is applied to control processes such as, but not limited to, calibration and ranging based on a fisheye camera.

[0034] refer to Figure 1 This application provides a target detection method based on a fisheye camera, the target detection method including:

[0035] Step 1: Obtain the first transformation relationship between the camera coordinate system and the world coordinate system of the fisheye camera, and the second transformation relationship between the camera coordinate system and the pixel coordinate system of the fisheye camera;

[0036] Step 2: Based on the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to the set plane in the world coordinate system, determine the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system.

[0037] In the above scheme, the first transformation relationship between the camera coordinate system and the world coordinate system of the fisheye camera, and the second transformation relationship between the camera coordinate system and the pixel coordinate system of the fisheye camera are obtained first. Then, based on the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system, the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system are determined. Compared with the existing technology that requires the optical center of the fisheye camera to be on the same horizontal plane as the center of the target object, this application, after completing the internal and external parameter calibration, determines the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system based on the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system. This eliminates the need for the optical center of the fisheye camera to be on the same horizontal plane as the center of the target object, supports different installation angles of the fisheye camera, reduces installation requirements and difficulty, expands the application scenarios, and the determined coordinates of the target point in the world coordinate system are closer to the position of the real scene. The following is a detailed description of each step with reference to the accompanying drawings.

[0038] First, refer to Figure 1 This involves obtaining the first transformation relationship between the fisheye camera's camera coordinate system and the world coordinate system, and the second transformation relationship between the fisheye camera's camera coordinate system and the pixel coordinate system. Specifically, after fixing the fisheye camera's mounting position, intrinsic and extrinsic parameter calibrations are performed on the fisheye camera to obtain the first transformation relationship between the fisheye camera's camera coordinate system and the world coordinate system, as well as the second transformation relationship between the fisheye camera's camera coordinate system and the pixel coordinate system.

[0039] There are various methods for obtaining the first transformation relationship between the camera coordinate system and the world coordinate system of a fisheye camera. One implementation method is described below as an example.

[0040] For example, when determining the first transformation relationship between the camera coordinate system and the world coordinate system of a fisheye camera, the following steps can be taken: First, obtain the coordinates of at least four calibration point pairs. Each calibration point pair includes: a pixel coordinate system calibration point and a world coordinate system calibration point corresponding to the pixel coordinate system calibration point. The coordinates of the pixel coordinate system calibration point are the coordinates of the pixel coordinate system calibration point in the pixel coordinate system, and the coordinates of the world coordinate system calibration point are the coordinates of the world coordinate system calibration point in the world coordinate system. All world coordinate system calibration points are located in a set plane. Then, based on the second transformation relationship of the coordinates of at least four calibration point pairs, determine the rotation vector and translation vector between the camera coordinate system and the world coordinate system. Then, based on the rotation vector and translation vector, obtain the extrinsic parameter matrix of the fisheye camera, and use the extrinsic parameter matrix as the first transformation relationship. This method does not require obtaining the position of the optical center of the fisheye camera in the world coordinate system. Instead, it indirectly reflects the position of the optical center of the fisheye camera in the world coordinate system through the coordinates of the calibration point pairs in their respective coordinate systems, thereby obtaining the rotation vector and translation vector between the camera coordinate system and the world coordinate system.

[0041] In some embodiments, when obtaining the second transformation relationship through intrinsic parameter calibration, the second transformation relationship may include the intrinsic parameter matrix and distortion coefficients of the fisheye camera, wherein the distortion coefficients can be 4 bits. The second transformation relationship can be obtained by calibrating the intrinsic parameters of the fisheye camera. The intrinsic parameters of the fisheye camera can be calibrated using a checkerboard pattern. The fisheye camera needs to capture multiple checkerboard photos covering various angles with clear grid points. Then, the intrinsic parameters of the fisheye camera are calibrated using Matlab, resulting in a 3×3 intrinsic parameter matrix and 4-bit distortion coefficients. Matlab is a programming and numerical computation platform that supports data analysis, algorithm development, and modeling.

[0042] Specifically, a checkerboard pattern with clear grid points and a smooth surface is needed as an auxiliary tool. Multiple photos of the checkerboard pattern at different positions and angles are taken using a fisheye camera, ensuring that the corner points of the checkerboard pattern are clear and complete in the images. The checkerboard pattern size information, including the number of squares and its length and width, is then imported along with the captured images into Matlab for intrinsic parameter calibration, resulting in a 3×3 intrinsic parameter matrix. And 4-bit distortion coefficient.

[0043] For example, the following describes a method for determining a first transformation relationship between the camera coordinate system and the world coordinate system of a fisheye camera.

[0044] First, after the vehicle comes to a complete stop, select four or more calibration point pairs. Measure the 3D coordinates of the world coordinate system calibration point in each calibration point pair within the world coordinate system, thus obtaining the coordinates of each world coordinate system calibration point in the world coordinate system. During selection, ensure that all world coordinate system calibration points are located within the designated plane. Next, use a fisheye camera to photograph all world coordinate system calibration points, obtaining an image of the corresponding pixel coordinate system calibration point for each world coordinate system calibration point on the captured image. Then, measure the 2D coordinates of the pixel coordinate system calibration point in each calibration point pair within the pixel coordinate system, thus obtaining the coordinates of each pixel coordinate system calibration point in the pixel coordinate system.

[0045] Next, based on the second transformation relationship of the coordinates of the four calibration point pairs, the rotation and translation vectors between the camera coordinate system and the world coordinate system are determined. Specifically, the point-to-point coordinates of the 3D and 2D coordinates of all calibration point pairs, the intrinsic parameter matrix, and the distortion coefficients are used for extrinsic parameter calibration to obtain the rotation and translation vectors. Then, based on the rotation and translation vectors, the extrinsic parameter matrix of the fisheye camera is obtained, and the extrinsic parameter matrix is ​​used as the first transformation relationship.

[0046] The aforementioned extrinsic parameter calibration process requires four or more calibration point pairs. The world coordinate system calibration point in each calibration point pair is located on the image captured by the fisheye camera, and each pixel coordinate system calibration point on the fisheye camera image corresponds to a world coordinate system calibration point. It is necessary to measure the pixel coordinates (x, y) of the pixel coordinate system calibration point in each calibration point pair, as well as the coordinates (Xw, Yw, Zw) of the world coordinate system calibration point in each calibration point pair. Using the coordinates of multiple calibration point pairs, the intrinsic parameter matrix of the fisheye camera, and the distortion coefficients in the second transformation relationship, the rotation vector rvec and translation vector tvec from the world coordinate system to the camera coordinate system are obtained, leading to the extrinsic parameter matrix and the first transformation relationship.

[0047] For example, when calibrating external parameters, refer to Figure 2A fixed fisheye camera is required. When the plane is set to the ground where the vehicle is located, calibration can be performed by manually selecting pixels after taking a picture of the ground. First, the position of the fisheye camera needs to be determined. In intelligent driving vehicles, fisheye cameras are often used to capture surround view images and are installed in the four directions of the vehicle: front, rear, left, and right. In some embodiments, the center of the rear axle of the vehicle can be used as the origin of the world coordinate system (or vehicle coordinate system). After determining the position of the fisheye camera, the 3D points with known coordinates on the ground in the world coordinate system are determined. At least four world coordinate system calibration points are required, and all four world coordinate system calibration points must appear in the image taken by the fisheye camera. Then, the 2D coordinates of the four pixel coordinate system calibration points in the pixel coordinate system of the image are manually selected using the original fisheye image. Each pixel calibration point pair has a one-to-one correspondence between the world coordinate system calibration point and the pixel coordinate system calibration point. Using each calibration point pair, consisting of a 2D point in the pixel coordinate system and a 3D point in the world coordinate system, four calibration point pairs are formed. Together with the intrinsic parameter matrix and distortion coefficients, a 3×1 rotation vector rvec and a 3×1 translation vector tvec are obtained from the world coordinate system to the camera coordinate system. The rotation vector rvec can be used to obtain the 3×3 rotation matrix R using the Rodriguez formula shown below.

[0048]

[0049] Where I is the identity matrix, n is the unit vector of the rotation vector, and θ is the magnitude of the rotation vector. tvec is the translation vector, representing the relative displacement between the origin of the world coordinate system and the origin of the camera coordinate system. Let T = tvec, then the extrinsic parameter matrix of the fisheye camera is... Then, the extrinsic parameter matrix can be used as the first transformation relation.

[0050] It should be understood that, in addition to the methods shown above, other methods may be used to determine the first transformation relationship between the camera coordinate system and the world coordinate system of a fisheye camera.

[0051] For example, when obtaining the first transformation relationship between the camera coordinate system and the world coordinate system of a fisheye camera, the following steps can also be taken: First, obtain the position of the optical center of the fisheye camera in the world coordinate system; then, based on the position and deflection angle of the optical center, determine the translation vector between the camera coordinate system and the world coordinate system, and determine the deflection angle of one of the coordinate axes in the rotation vector. For example, the deflection angle between the Z-axis of the camera coordinate system and the Z-axis of the world coordinate system can be determined. Specifically, when obtaining the position of the optical center of the fisheye camera in the world coordinate system, its coordinate position in the world coordinate system can be obtained. The optical center of the fisheye camera is the origin of the camera coordinate system. Therefore, based on the coordinate position of the optical center of the fisheye camera and the origin position of the world coordinate system, the translation vector between the camera coordinate system and the world coordinate system can be determined. Then, by using the directions of the other two coordinate axes defined in the camera coordinate system and the world coordinate system, the deflection angles of the other two coordinate axes in the rotation vector can be determined, thus obtaining the complete rotation vector. For example, based on the directions of the X and Y axes defined in the camera coordinate system and the world coordinate system respectively, the deflection angles between the X-axis of the camera coordinate system and the X-axis of the world coordinate system, and the deflection angles between the Y-axis of the camera coordinate system and the Y-axis of the world coordinate system, can be determined to obtain a complete rotation vector. Then, based on the rotation vector and translation vector, the extrinsic parameter matrix of the fisheye camera is obtained, and this extrinsic parameter matrix is ​​used as the first transformation relation. That is, the final first transformation relation includes the extrinsic parameter matrix to complete the extrinsic parameter calibration of the fisheye camera.

[0052] In some embodiments, the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system can be obtained after the fisheye camera's mounting position is fixed. The optical axis of the fisheye camera is a directed line drawn from the center of the fisheye camera's lens, with the optical center as the starting point. The deflection angle refers to the angle between the optical axis and the set plane (a right angle). The set plane can be any plane in the world coordinate system. For example, the set plane can be parallel to the first and second coordinate axes in the world coordinate system. As an example, the first and second coordinate axes can be the Xw and Yw coordinate axes in the world coordinate system, respectively, meaning the set plane is a plane perpendicular to the Zw coordinate axis. In some embodiments, the set plane can be the XwOwYw plane in the world coordinate system, where Ow is the origin of the world coordinate system. Of course, in other embodiments, when the fisheye camera is applied to a vehicle, the ground where the vehicle is located can be used as the set plane. In some embodiments, the center of the vehicle's rear axle can be used as the origin of the world coordinate system (or vehicle coordinate system).

[0053] For example, refer to Figure 4 and Figure 5 The optical axis of a fisheye camera can be considered as the Zc coordinate axis in the camera coordinate system. (Reference) Figure 4 and Figure 5 The deflection angle can be represented by θ. For example, the deflection angle θ can be the angle between the Zc coordinate axis in the camera coordinate system and the XwOwYw plane in the world coordinate system. In some embodiments, the deflection angle θ between the optical axis of the fisheye camera and the set plane can be no less than 0° and no greater than 90°, meaning the fisheye camera is positioned relative to the set plane and facing it. Specifically, the deflection angle can be any value between 0° and 90°, such as 0°, 30°, 45°, 60°, 75°, or 90°. When the set plane is the ground where the vehicle is located, the fisheye camera is positioned facing the ground.

[0054] For example, the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system can be obtained in the following manner. For example, the deflection angle is calculated based on a first transformation relationship. Specifically, the deflection angle of the fisheye camera's optical axis relative to the set plane in the world coordinate system can be determined according to the first transformation relationship. Of course, in addition to the methods shown above, other methods can also be used. For example, the deflection angle can also be obtained in advance based on the fisheye camera's installation position, or even by measurement.

[0055] Next, refer to Figure 1 Based on the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system, the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system are determined. In other words, the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system are used to determine the corresponding coordinates of the target point in the world coordinate system in the image captured by the fisheye camera. Specifically, there are various methods to determine the coordinates of the target point in the world coordinate system in the image captured by the fisheye camera using the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system.

[0056] For example, when determining the corresponding coordinates of a target point in the world coordinate system of an image captured by a fisheye camera based on the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system, the following method can be used: Based on the image captured by the fisheye camera, obtain the first coordinates of the target point in the pixel coordinate system; based on the first coordinates, the second transformation relationship, and the deflection angle, determine the second coordinates of the target point in the camera coordinate system; based on the second coordinates and the first transformation relationship, determine the third coordinates of the target point in the world coordinate system. That is, in the process of obtaining the second coordinates of the target point in the camera coordinate system based on the first coordinates of the target point in the pixel coordinate system, not only the second transformation relationship but also the deflection angle are utilized, thereby enabling a more accurate determination of the second coordinates of the target point in the camera coordinate system. Furthermore, it eliminates the need for the optical center of the fisheye camera to be on the same horizontal plane as the center of the target object, supports different installation angles for the fisheye camera, reduces installation requirements and difficulty, and expands the application scenarios.

[0057] When determining the second coordinates of the target point in the camera coordinate system based on the first coordinates, the second transformation relationship, and the deflection angle, various methods can be used. For example, the following method can be employed: First, obtain the vertical distance of the target point from the set plane in the world coordinate system; then, determine the third coordinates of the target point in the camera coordinate system based on the first coordinates, the second transformation relationship, the deflection angle θ, and the vertical distance.

[0058] In an image captured by a fisheye camera with a fixed position, a target point can be selected, and its first coordinates in the pixel coordinate system can be obtained. Then, the vertical distance of the target point from the set plane in the world coordinate system is also obtained. Since the first transformation relationship in the aforementioned calibration process includes the vertical distance from the origin of the camera coordinate system to the set plane, the distance H between the target point and the origin of the camera coordinate system in the direction perpendicular to the set plane can be determined based on the vertical distance from the origin of the camera coordinate system to the set plane and the vertical distance from the target point to the set plane. Then, the second coordinates of the target point in the camera coordinate system can be determined based on the first coordinates, the second transformation relationship, the deflection angle θ, and the vertical distance H. In some embodiments, when the target point is located on the set plane, the vertical distance from the target point to the set plane is equal to the vertical distance from the origin of the camera coordinate system to the set plane. For example, the plane can be defined as the XwOwYw plane in the world coordinate system, and the target point can be a point on the XwOwYw plane in the world coordinate system. This facilitates the accurate acquisition of the target point with the first coordinate in the pixel coordinate system and its corresponding third coordinate in the world coordinate system. This also facilitates the calculation of parameters such as the distance between different points in the world coordinate system.

[0059] It needs to be explained that the reference Figure 3 In this embodiment, when determining the second coordinate Pc of the target point in the camera coordinate system based on the deflection angle and the second transformation relationship, the Zc in the second coordinate Pc is mainly solved using the deflection angle θ and the vertical distance H. In the prior art, because the optical axis of the fisheye camera is parallel to the ground, the Zc of all points is a fixed value. Therefore, compared with the prior art, this application does not require the optical center of the fisheye camera to be on the same horizontal plane as the center of the target object, supports different installation angles of the fisheye camera, reduces installation requirements and difficulty, and has more application scenarios.

[0060] For example, refer to Figure 2 The target detection method may further include: using a target detection algorithm to detect target objects in an image, and determining target points based on the target objects. That is, in terms of how to obtain target points from images captured by a fisheye camera, this application also discloses a method for selecting target points in some embodiments, specifically using a target detection algorithm to detect target objects in an image, and determining target points based on the target objects in the image. The detection method can employ various approaches, several of which are exemplarily described below.

[0061] For example, when using an object detection algorithm to detect target objects in an image and determining target points based on the target objects, the following steps can be taken: First, the target object is bounded in the image using a detection box generated by the object detection algorithm; then, the target points of the target object are determined based on the detection box. Specifically, the target object is bounded in the image using a detection box generated by the object detection algorithm. This target object can be a regularly shaped object, such as a rectangle, a circle, or other regularly shaped objects. In other embodiments, the target object can also be an irregularly shaped object. For example, the target object can be, but is not limited to, objects such as people, telephone poles, vehicles, and traffic cones; it is not required that the target object be a cube. The shape of the detection box can be, but is not limited to, a rectangle. In some embodiments, as the number of target object categories increases, other categories can be added through training to optimize the object detection algorithm. The number of categories is increased through model training, and accuracy is improved. Operations such as distance measurement can be performed on targets of interest.

[0062] After defining the target object, a target point representing the object's position can be selected based on the detection box. The position of the target point represents the target object's position. Various methods can be used for selection. For example, when determining the target point of the target object based on the detection box, the midpoint of the bottom edge of the detection box can be used as the target point. It should be understood that the method of determining the target point of the target object based on the detection box is not limited to the methods shown above; other methods can also be used. For example, the bottom edge endpoint of the detection box or the center point of the detection box can also be selected as the target point.

[0063] For example, refer to Figure 2 The target detection method may further include: determining the distance between a target point and a set point based on a third coordinate and the set point's set coordinates in the world coordinate system. In some embodiments, the set point may be a pre-calibrated reference point. For example, the set point may be the origin of the world coordinate system or the optical center of a fisheye camera, and its coordinates in the world coordinate system are known set coordinates. In other embodiments, the set point may be another target point picked up from an image captured by a fisheye camera, and the coordinate information of the target point in the world coordinate system is obtained by using the same transformation relationship described above.

[0064] For example, when determining the distance between the target point and the set point based on the third coordinate and the set point's set coordinates in the world coordinate system, this distance can be the distance of a direct line connecting the target point and the set point. The distance between the target point and the set point can be determined based on the third coordinate and the set point's set coordinates in the world coordinate system; specifically, the calculation can be done by finding the distance between two points in the same coordinate system.

[0065] In other embodiments, when determining the distance between the target point and the set point based on the third coordinates and the set point's set coordinates in the world coordinate system, the first projection point and the second projection point of the target point and the set point, respectively, projected vertically onto the set plane can be determined based on the third coordinates and the set coordinates, and the distance between the first projection point and the second projection point can be determined. That is, first, the first projection point of the target point projected vertically onto the set plane and the second projection point of the set point projected vertically onto the set plane are determined. Then, the distance between the first projection point and the second projection point is calculated based on the third coordinates and the set coordinates. When the set plane is the plane where the vehicle is located, the distance between the first projection point and the second projection point represents the horizontal distance between the target point as a ground point and the set point, thus facilitating the calculation of the distance between the target object and the vehicle.

[0066] For example, after calibration, a fisheye camera can be used for target ranging. The pose of the fisheye camera, i.e., its installation position and angle, must be consistent with the extrinsic parameter calibration. The following example uses the setting plane as the ground where the vehicle is located, the target object as an object on the ground, the target point as the point where the target object contacts the ground, the distance between the setting plane and the optical center of the fisheye camera as H, and the deflection angle between the optical axis of the fisheye camera and the setting plane as θ. This example illustrates the principles of the relevant transformations of the target point's coordinates between the pixel coordinate system, the camera coordinate system, and the world coordinate system.

[0067] The formula for converting a 3D point in world coordinates to a 2D point in pixel coordinates is:

[0068]

[0069] Furthermore, the transformation formula from point Pw(Xw,Yw,Zw) in the world coordinate system to point Pc(Xc,Yc,Zc) in the camera coordinate system is:

[0070]

[0071] First, the distortion-free image captured by the fisheye camera is processed using an object detection algorithm. A detection box is output for the target object of interest. The lower midpoint of the detection box is used as the ground point on the image, resulting in a 2D point in pixel coordinates. Figure 3 Point P(x,y) is shown.

[0072] From formulas (1) and (2), we can obtain that

[0073]

[0074] Right now,

[0075]

[0076] As shown in formula (3), transforming a point P(x,y) in the pixel coordinate system to a point Pc(Xc,Yc,Zc) in the camera coordinate system requires knowing the projection Zc of point Pc onto the Z-axis of the camera coordinate system and the camera's intrinsic parameter matrix K. Since the 2D point P(x,y) of the target object in the pixel coordinate system and the intrinsic parameter matrix K of the fisheye camera are known, let... but In (5), the left-hand side of the equation is known. Now, we need to solve for Pc(Xc,Yc,Zc).

[0077] According to formula (5),

[0078]

[0079] It can be obtained by deformation.

[0080]

[0081] like Figure 4 As shown, the solution for Zc is discussed in two cases:

[0082] ① If the 3D point is to the left of the intersection of the camera's optical axis and the world coordinate system's Yw axis, and ΔAOwC ~ ΔBPcC, according to the principle of similar triangles, we have

[0083]

[0084] Given the camera height H and the camera mounting angle θ, we have:

[0085]

[0086] Substituting formula (9) into formula (8) yields the following:

[0087]

[0088] After simplification, substituting formula (7) into the equation yields the following result:

[0089]

[0090] ②If the 3D point is to the right of the intersection of the camera's optical axis and the world coordinate system's Yw axis, then for the area S of ΔCOcPc′, we have:

[0091]

[0092] Therefore,

[0093] |OcC||P'D|=H|CP′| (11)

[0094] P'(Xc',Yc',Zc') lies on the negative half of the camera's Yc axis, so Yc' is negative. Substituting the camera height H and angle θ into formula (11) yields the following result.

[0095]

[0096] After simplification, substituting formula (7) into the equation yields the following result:

[0097]

[0098] In summary, formula (10) applies to all points in the camera coordinate system.

[0099] A point (Xc, Yc, Zc) in the camera coordinate system is transformed into the world coordinate system with Ow as the origin, such as... Figure 4 As shown, according to formula (2), we have,

[0100]

[0101] extrinsic parameter matrix The location of the target object's grounding 3D point in the world coordinate system, obtained through calibration, is (Xw, Yw, Zw) when substituted into the above formula. The distance L from the world coordinate system is also given.

[0102]

[0103] In the ranging method described above, the target point is directly selected from the image captured by the fisheye camera, eliminating the need for contour extraction and saving this step. The distance from the target object to the set point is calculated using the midpoint of the lower edge of the detection box as the target point. This simplifies the algorithm while maintaining ranging accuracy. Specifically, by directly using the original fisheye image for target detection and ranging, the stretching and cropping of objects by the distortion correction image is avoided, reducing image distortion correction steps and minimizing the impact of intrinsic parameters on ranging errors, while ensuring the detection range and the number of detected target objects. Furthermore, the center of the target object does not need to be on the same horizontal plane as the optical center of the fisheye camera, solving the ranging problem when the target object's center is at different positions on the camera's optical axis. Using the target's ground point as the target point for distance calculation broadens its applicability.

[0104] As shown in the ranging section above, the process begins with capturing an image using a fisheye camera. This distorted original image is then fed into a target detection algorithm model for target detection. This model can be replaced with any suitable target detection model based on hardware platform or system requirements. After target detection, a 2D bounding box is obtained. The midpoint of the lower bounding box is used as the first coordinate of the 2D grounding point (target point) of the target object in the image. Using an intrinsic parameter matrix and distortion coefficients, the 2D target point is transformed to the camera coordinate system, and then further transformed to the world coordinate system via an extrinsic parameter matrix. This yields the 3D grounding point of the target object in the world coordinate system, which is the third coordinate of the target point in the world coordinate system. Finally, the distance from the target object to the origin of the world coordinate system or the optical center of the fisheye camera is calculated based on the coordinates of the 3D grounding point.

[0105] In the various embodiments shown above, a first transformation relationship between the camera coordinate system and the world coordinate system of the fisheye camera, and a second transformation relationship between the camera coordinate system and the pixel coordinate system of the fisheye camera are first obtained. Then, based on the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system, the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system are determined. Compared with the prior art, which requires the optical center of the fisheye camera to be on the same horizontal plane as the center of the target object, this application, after completing the calibration of internal and external parameters, determines the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system based on the first transformation relationship, the second transformation relationship, and the deflection angle of the fisheye camera's optical axis relative to a set plane in the world coordinate system. This eliminates the need for the optical center of the fisheye camera to be on the same horizontal plane as the center of the target object, supports different installation angles of the fisheye camera, reduces installation requirements and difficulty, expands the application scenarios, and the determined coordinates of the target point in the world coordinate system are closer to the position of the real scene.

[0106] Compared to related technologies that are only applicable to distance measurement between a fisheye camera and a planar checkerboard pattern, and whose checkerboard size is known, making them unsuitable for measuring the distance to unknown-sized target objects in autonomous vehicles, the ranging method provided in some embodiments of this application is applicable to ranging different types of objects, and the object size does not need to be measured in advance, making it more practical and applicable to a wider range of situations.

[0107] Compared to related technologies that require the optical center of the fisheye camera lens to be on the same horizontal plane as the center of the object, and also require contour recognition and feature point extraction of the framed image, the applicable ranging conditions are more stringent, the target object height must be a fixed value, and the algorithm steps are more numerous. The ranging method provided in some embodiments of this application is applicable to common road surface objects, has no specific height requirements, and directly uses the midpoint of the lower edge of the target detection algorithm's output frame as the object's ground point for ranging, ensuring ranging accuracy while simplifying the algorithm.

[0108] Compared to related technologies that use fisheye distortion-corrected images for target ranging, the edges of the distortion-corrected image stretch the target object, affecting the detection effect. Furthermore, scaling and cropping the image reduces the detection range and the number of target objects, further impacting detection performance. In some embodiments of this application, the original fisheye image is used for target detection and ranging, which reduces the impact of intrinsic parameter errors while ensuring the detection range and the number of detected target objects.

[0109] Compared with the prior art, the above embodiments of this application have the following differences:

[0110] 1. Existing technology requires that the optical axis of a fisheye camera be horizontal to the ground, but in reality, fisheye cameras are installed at a certain angle. Some embodiments of this application consider different uses and installation angles of fisheye cameras, and can be used for non-eye-level cameras, that is, when the optical axis of the fisheye camera is not parallel to the horizontal plane.

[0111] 2. Existing technologies require the center of the target object to be on the same horizontal plane as the camera's optical center. However, in practical applications, the object may be located at different positions along the camera's optical axis, making it difficult to ensure that the center is always on the camera's optical axis. In some embodiments of this application, the center of the target object does not need to be on the same horizontal plane as the fisheye lens's optical center, and this applies to all grounded target objects.

[0112] 3. Existing technologies require that the target object be a cube, which limits the scope of application. In some embodiments of this application, the applicable target object can be a non-cubic grounded target such as a person, vehicle, or chair, and other target categories can be added through training.

[0113] 4. Existing technologies use ResNet for object detection, perform image processing on the obtained bounding box to obtain the target contour map, then perform contour tracking, obtain the feature points of the target object based on the feature points of the contour, and calculate the distance based on the feature points. This method is too complicated. In some embodiments of this application, the object detection algorithm is used to output the detection box of the target object in real time, and the distance from the midpoint of the ground edge of the detection box to the camera is measured, eliminating the step of calculating the feature points;

[0114] 5. Existing technologies use fisheye distortion-corrected images for target ranging. The edges of the distortion-corrected image stretch the target object, affecting the detection effect. After scaling and cropping the image, the detection range and the number of target objects are reduced, affecting the detection effect. In some embodiments of this application, the original fisheye image is used directly for target detection and ranging, which can reduce the influence of intrinsic parameters on ranging error, while ensuring the detection range and the number of detected target objects.

[0115] Furthermore, this application also provides a target detection device based on a fisheye camera. The target detection device based on a fisheye camera includes a storage medium and a processor. The storage medium stores a computer program that is executed by the processor. When the computer program is executed by the processor, the processor performs any of the above-mentioned target detection methods based on a fisheye camera.

[0116] Figure 6 A schematic block diagram of a target detection device 100 based on a fisheye camera according to an embodiment of this application is shown. Figure 6 As shown, the target detection device 100 based on a fisheye camera according to an embodiment of this application may include a storage medium 110 and a processor 120. The storage medium 110 stores a computer program executed by the processor 120. When the computer program is executed by the processor 120, the processor 120 performs the target detection method based on a fisheye camera according to the embodiments of this application described above. Those skilled in the art can understand the specific operation of the deployment device of the target detection device 100 based on the embodiments of this application in conjunction with the foregoing content. For the sake of brevity, it will not be described in detail here.

[0117] The storage medium 110 may include, for example, a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. A computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0118] In addition, this application also provides a vehicle, which includes: a vehicle body, a fisheye camera mounted on the vehicle body, and any of the above-described target detection devices based on the fisheye camera. Specifically, the vehicle can be, but is not limited to, an electric vehicle, a hybrid vehicle, or a gasoline-powered vehicle; that is, any vehicle equipped with a fisheye camera can be used as the vehicle in this application embodiment. The fisheye camera can be mounted around the vehicle body to detect target objects around the vehicle body, and then the distance to the detected target objects can be measured using the method shown in the above embodiments.

[0119] The present invention has been described through the above embodiments. However, it should be understood that the above embodiments are for illustrative purposes only and are not intended to limit the invention to the scope of the described embodiments. Furthermore, those skilled in the art will understand that the present invention is not limited to the above embodiments, and many more variations and modifications can be made based on the teachings of the present invention, all of which fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A target detection method based on a fisheye camera, characterized in that, include: Obtain the first transformation relationship between the camera coordinate system and the world coordinate system of the fisheye camera, and the second transformation relationship between the camera coordinate system and the pixel coordinate system of the fisheye camera; Based on the first transformation relationship, the second transformation relationship, and the deflection angle of the optical axis of the fisheye camera relative to the set plane in the world coordinate system, the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system are determined; The deflection angle is obtained in advance based on the installation position of the fisheye camera, or calculated based on the first conversion relationship.

2. The target detection method as described in claim 1, characterized in that, The angle of deflection between the optical axis of the fisheye camera and the set plane is not less than 0° and not greater than 90°.

3. The target detection method as described in claim 1, characterized in that, The step of determining the corresponding coordinates of a target point in the world coordinate system in the image captured by the fisheye camera, based on the first transformation relationship, the second transformation relationship, and the deflection angle of the optical axis of the fisheye camera relative to a set plane in the world coordinate system, includes: Based on the image captured by the fisheye camera, obtain the first coordinates of the target point in the pixel coordinate system; Based on the first coordinates, the second transformation relationship, and the deflection angle, determine the second coordinates of the target point in the camera coordinate system; Based on the second coordinates and the first transformation relationship, the third coordinates corresponding to the target point in the world coordinate system are determined.

4. The target detection algorithm as described in claim 3, characterized in that, Determining the second coordinates of the target point in the camera coordinate system based on the first coordinates, the second transformation relationship, and the deflection angle includes: Obtain the vertical distance of the target point from the set plane in the world coordinate system; Based on the first coordinates, the second transformation relationship, the deflection angle, and the vertical distance, the second coordinates corresponding to the target point in the camera coordinate system are determined.

5. The target detection method as described in claim 3, characterized in that, Also includes: The target object in the image is detected using an object detection algorithm, and the target point is determined based on the target object.

6. The target detection method as described in claim 5, characterized in that, The step of using an object detection algorithm to detect target objects in the image and determining the target points based on the target objects includes: The target object is defined on the image using the detection box generated by the target detection algorithm. The target point of the target object is determined based on the detection frame.

7. The target detection method as described in claim 6, characterized in that, Determining the target point of the target object based on the detection frame includes: The midpoint of the lower edge of the detection frame is taken as the target point.

8. The target detection method as described in claim 7, characterized in that, Also includes: The distance between the target point and the set point is determined based on the third coordinate and the set point's set coordinates in the world coordinate system.

9. The target detection method as described in claim 8, characterized in that, Determining the distance between the target point and the set point based on the third coordinate and the set point's set coordinates in the world coordinate system includes: Based on the third coordinate and the set coordinate, determine the first projection point and the second projection point of the target point and the set point respectively projected vertically onto the set plane, and determine the distance between the first projection point and the second projection point.

10. The target detection method as described in claim 8, characterized in that, The set point is the origin of the world coordinate system or the optical center of the fisheye camera.

11. The target detection method as described in claim 1, characterized in that, The defined plane is parallel to both the first and second coordinate axes in the world coordinate system.

12. The target detection method as described in claim 1, characterized in that, The step of obtaining the first transformation relationship between the camera coordinate system and the world coordinate system of the fisheye camera includes: Obtain the coordinates of at least four calibration point pairs, each calibration point pair including: a pixel coordinate system calibration point and a world coordinate system calibration point corresponding to the pixel coordinate system calibration point; wherein, the coordinates of the pixel coordinate system calibration point are the coordinates of the pixel coordinate system calibration point in the pixel coordinate system, the coordinates of the world coordinate system calibration point are the coordinates of the world coordinate system calibration point in the world coordinate system, and all the world coordinate system calibration points are located in the set plane; Based on the coordinates of at least four calibration point pairs and the second transformation relationship, determine the rotation vector and translation vector between the camera coordinate system and the world coordinate system; The extrinsic parameter matrix of the fisheye camera is obtained based on the rotation vector and the translation vector, and the extrinsic parameter matrix is ​​used as the first transformation relationship.

13. The target detection method as described in claim 1, characterized in that, The second transformation relationship includes the intrinsic parameter matrix and distortion coefficients of the fisheye camera.

14. A target detection device based on a fisheye camera, characterized in that, include: A storage medium and a processor, wherein the storage medium stores a computer program executed by the processor, the computer program, when executed by the processor, causes the processor to perform the target detection method based on a fisheye camera as described in any one of claims 1 to 13.

15. A vehicle, characterized in that, include: Vehicle body; A fisheye camera mounted on the vehicle body; The target detection device based on a fisheye camera as described in claim 14 is installed on the vehicle body.

Images