Target detection method and device based on fisheye camera, and vehicle

Abstract

The invention discloses a target detection method and device based on a fisheye camera, and a vehicle. The target detection method comprises the following steps: obtaining a first conversion relation between a camera coordinate system of the fisheye camera and a world coordinate system, and a second conversion relation between the camera coordinate system of the fisheye camera and a pixel coordinate system; and according to the first conversion relation, the second conversion relation and a deflection angle of an optical axis of the fisheye camera relative to a set plane in the world coordinate system, determining a corresponding coordinate of a target point in an image shot by the fisheye camera in the world coordinate system. The optical center of the fisheye camera and the center of the target detection object do not need to be located on the same horizontal plane, different installation angles of the fisheye camera are supported, installation requirements and installation difficulty are reduced, and more application scenes are achieved.

Description

Technical Field

[0001] The present application relates to the field of cameras, and more specifically to a target detection method based on a fisheye camera and a device and vehicle thereof. Background Art

[0002] Fisheye cameras, with their large field of view, are used in various occasions such as but not limited to vehicles and surveillance. They can be used for, but not limited to, target distance measurement. In the related art, when installing a fisheye camera, it is necessary to ensure that the optical center of the fisheye camera and the center of the target detection object are on the same horizontal plane, which leads to stricter installation requirements and increases the difficulty of installation. Summary of the invention

[0003] The present application is proposed to solve at least one of the above problems. According to the first aspect of the present application, a target detection method based on a fisheye camera is provided, the target detection method comprising: obtaining a first conversion relationship between a camera coordinate system of the fisheye camera and the world coordinate system, and a second conversion relationship between the camera coordinate system of the fisheye camera and a pixel coordinate system; determining the corresponding coordinates of a target point in the image taken by the fisheye camera in the world coordinate system according to the first conversion relationship, the second conversion relationship, and the deflection angle of the optical axis of the fisheye camera relative to a set plane in the world coordinate system.

[0004] In one embodiment of the present application, the target detection method further includes: the deflection angle is pre-calibrated according to the installation position of the fisheye camera; or, the deflection angle is calculated according to the first conversion relationship.

[0005] In one embodiment of the present application, a deflection angle between the optical axis of the fisheye camera and the setting plane is not less than 0° and not greater than 90°.

[0006] In one embodiment of the present application, the corresponding coordinates of a target point in the image taken by the fisheye camera in the world coordinate system are determined according to the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to a set plane in the world coordinate system, including: obtaining a first coordinate corresponding to the target point in the pixel coordinate system according to the image taken by the fisheye camera; determining a second coordinate corresponding to the target point in the camera coordinate system according to the first coordinate, the second conversion relationship and the deflection angle; and determining a third coordinate corresponding to the target point in the world coordinate system according to the second coordinate and the first conversion relationship.

[0007] In one embodiment of the present application, determining the second coordinate corresponding to the target point in the camera coordinate system based on the first coordinate, the second conversion relationship, and the deflection angle includes: obtaining the vertical distance of the target point from the set plane in the world coordinate system; determining the second coordinate corresponding to the target point in the camera coordinate system based on the first coordinate, the second conversion relationship, the deflection angle, and the vertical distance.

[0008] In one embodiment of the present application, the target detection method further includes: using a target detection algorithm to detect the target object on the image, and determining the target point according to the target object.

[0009] In one embodiment of the present application, the use of a target detection algorithm to detect the target object on the image and determining the target point based on the target object includes: using a detection frame of the target detection algorithm to frame the target object on the image; and determining the target point of the target object based on the detection frame.

[0010] In one embodiment of the present application, determining the target point of the target object according to the detection frame includes: taking the midpoint of the lower edge of the detection frame as the target point.

[0011] In one embodiment of the present application, the target detection method further includes: determining the distance between the target point and the set point according to the third coordinate and the set coordinates of the set point in the world coordinate system.

[0012] In one embodiment of the present application, determining the distance between the target point and the set point based on the third coordinate and the set coordinates of the set point in the world coordinate system includes: determining a first projection point and a second projection point where the target point and the set point are respectively vertically projected on the set plane based on the third coordinate and the set coordinate, and determining the distance between the first projection point and the second projection point.

[0013] In one embodiment of the present application, the set point is the origin of the world coordinate system or the optical center of the fisheye camera.

[0014] In one embodiment of the present application, the setting plane is parallel to both the first coordinate axis and the second coordinate axis in the world coordinate system.

[0015] In one embodiment of the present application, determining the first transformation relationship between the camera coordinate system and the world coordinate system of the fisheye camera includes: obtaining the coordinates of at least four calibration point pairs, each calibration point pair including: a pixel coordinate system calibration point, and a world coordinate system calibration point corresponding to the pixel coordinate system calibration point; wherein the coordinates of the pixel coordinate system calibration point are the coordinates of the pixel coordinate system calibration point in the pixel coordinate system, the coordinates of the world coordinate system calibration point are the coordinates of the world coordinate system calibration point in the world coordinate system, and all of the world coordinate system calibration points are located in the set plane; determining the rotation vector and the translation vector between the camera coordinate system and the world coordinate system based on the coordinates of at least four of the four calibration point pairs and the second transformation relationship; obtaining the extrinsic parameter matrix of the fisheye camera based on the rotation vector and the translation vector, and using the extrinsic parameter matrix as the first transformation relationship.

[0016] In one embodiment of the present application, the second conversion relationship includes an intrinsic parameter matrix and distortion coefficients of the fisheye camera.

[0017] According to the second aspect of the present application, a target detection device based on a fisheye camera is also provided, and the target detection device includes: a storage medium and a processor, and the storage medium stores a computer program run by the processor, and when the computer program is run by the processor, the processor executes any one of the above-mentioned target detection methods based on a fisheye camera.

[0018] According to a third aspect of the present application, a vehicle is further provided, comprising: a vehicle body, a fisheye camera arranged on the vehicle body, and any one of the above-mentioned target detection devices based on the fisheye camera.

[0019] According to the target detection method of the fisheye camera provided by the embodiment of the present application, the device and vehicle thereof are provided, by first obtaining the first conversion relationship between the camera coordinate system of the fisheye camera and the world coordinate system, and the second conversion relationship between the camera coordinate system of the fisheye camera and the pixel coordinate system; then according to the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to the plane set in the world coordinate system, the corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system are determined. Compared with the installation requirement of the prior art that requires the optical center of the fisheye camera and the center of the target detection object to be in the same horizontal plane, the present application determines the corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system according to the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to the plane set in the world coordinate system after completing the internal and external parameter calibration, without making the optical center of the fisheye camera and the center of the target detection object in the same horizontal plane, supports different installation angles of the fisheye camera, reduces installation requirements and installation difficulty, and has more usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0021] Figure 1 This is a flow chart of a target detection method using a fisheye camera according to an embodiment of the present invention;

[0022] Figure 2 This is a flow chart of a method for detecting an object using a fisheye camera according to another embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the imaging principle of a fisheye camera according to an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of YOZ plane imaging of a fisheye camera in a world coordinate system is shown in one embodiment of the present invention;

[0025] Figure 5 A diagram showing the positional relationship between various coordinate systems according to an embodiment of the present invention;

[0026] Figure 6 The present invention is a schematic block diagram of a target detection device of a fisheye camera according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0028] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0029] It should be understood that the present invention can be implemented in different forms and should not be interpreted as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the disclosure thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0030] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present invention. When used herein, the singular forms "one", "an" and "said / the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, confirm the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0031] In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

[0032] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0033] First, the application scenario of the target detection method based on a fisheye camera illustrated in the example of the present application is introduced. The target detection method based on a fisheye camera is applied to control processes based on a fisheye camera such as but not limited to calibration and ranging.

[0034] refer to Figure 1 , an embodiment of the present application provides a target detection method based on a fisheye camera, the target detection method comprising:

[0035] Step 1: Obtain a first conversion relationship between the camera coordinate system of the fisheye camera and the world coordinate system, and a second conversion relationship between the camera coordinate system of the fisheye camera and the pixel coordinate system;

[0036] Step 2: Determine the corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system according to the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to the set plane in the world coordinate system.

[0037] In the above scheme, the first conversion relationship between the camera coordinate system of the fisheye camera and the world coordinate system, and the second conversion relationship between the camera coordinate system of the fisheye camera and the pixel coordinate system are first obtained; then the corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system are determined according to the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to the plane set in the world coordinate system. Compared with the installation requirement of the prior art that the optical center of the fisheye camera and the center of the target detection object need to be in the same horizontal plane, the present application determines the corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system according to the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to the plane set in the world coordinate system after completing the internal and external parameter calibration. It is not necessary to make the optical center of the fisheye camera and the center of the target detection object in the same horizontal plane, support different installation angles of the fisheye camera, reduce installation requirements and installation difficulty, use more scenarios, and the coordinates of the determined target point in the world coordinate system are closer to the position of the real scene. The above steps are described in detail below in conjunction with the accompanying drawings.

[0038] First, refer to Figure 1 , obtain the first conversion relationship between the camera coordinate system of the fisheye camera and the world coordinate system, and the second conversion relationship between the camera coordinate system of the fisheye camera and the pixel coordinate system. That is, after the installation position of the fisheye camera is fixed, the fisheye camera is calibrated with internal parameters and external parameters to obtain the first conversion relationship between the camera coordinate system of the fisheye camera and the world coordinate system, and the second conversion relationship between the camera coordinate system of the fisheye camera and the pixel coordinate system.

[0039] When specifically acquiring the first conversion relationship between the camera coordinate system of the fisheye camera and the world coordinate system, multiple methods may be used. An implementation method is exemplarily introduced as follows.

[0040] Exemplarily, when determining the first conversion relationship between the camera coordinate system and the world coordinate system of the fisheye camera, the following steps can be adopted: first, obtain the coordinates of at least four calibration point pairs, each calibration point pair includes: a pixel coordinate system calibration point, and a world coordinate system calibration point corresponding to the pixel coordinate system calibration point, wherein the coordinates of the pixel coordinate system calibration point are the coordinates of the pixel coordinate system calibration point in the pixel coordinate system, the coordinates of the world coordinate system calibration point are the coordinates of the world coordinate system calibration point in the world coordinate system, and all the world coordinate system calibration points are located in a set plane. Afterwards, according to the second conversion relationship of the coordinates of at least four calibration point pairs, the rotation vector and the translation vector between the camera coordinate system and the world coordinate system are determined. Then, according to the rotation vector and the translation vector, the external parameter matrix of the fisheye camera is obtained, and the external parameter matrix is ​​used as the first conversion relationship. This method does not need to obtain the position of the optical center of the fisheye camera in the world coordinate system, but indirectly reflects the position of the optical center of the fisheye camera in the world coordinate system through the coordinates of the calibration point pairs in their respective coordinate systems, thereby obtaining the rotation vector and the translation vector between the camera coordinate system and the world coordinate system.

[0041] In some embodiments, when the internal parameter calibration is performed to obtain the second conversion relationship, the second conversion relationship may include the internal parameter matrix and the distortion coefficient of the fisheye camera, wherein the number of bits of the distortion coefficient may be 4 bits. The second conversion relationship can be obtained by performing internal parameter calibration on the fisheye camera. The internal parameters of the fisheye camera can be calibrated with the help of a black and white chessboard. The fisheye camera needs to take multiple chessboard photos covering various angles and with clear grid points, and then use the Matlab tool to calibrate the internal parameters of the fisheye camera to obtain a 3×3 internal parameter matrix and a 4-bit distortion coefficient. Among them, Matlab is a programming and numerical computing platform that supports data analysis, algorithm development, and modeling.

[0042] Specifically, a chessboard with clear grid points and a flat surface is used as an auxiliary tool. A fisheye camera is used to take multiple photos of the chessboard at different positions and angles. The chessboard corners in the image must be clear and complete. The chessboard size information, including the number of grids and length and width information, is imported into the Matlab tool together with the captured images for internal parameter calibration to obtain a 3×3 internal parameter matrix. and 4-bit distortion coefficients.

[0043] Exemplarily, a method for determining a first conversion relationship between a camera coordinate system of a fisheye camera and a world coordinate system is introduced as follows.

[0044] First, the vehicle stops, select 4 or more calibration point pairs, measure the 3D coordinates of the world coordinate system calibration points in each calibration point pair in the world coordinate system, and obtain the coordinates of each world coordinate system calibration point in the world coordinate system. When selecting, it is necessary to ensure that all world coordinate system calibration points are located in the set plane. After that, use the installed fisheye camera to shoot all the world coordinate system calibration points to obtain the image of the pixel coordinate system calibration point corresponding to each world coordinate system calibration point on the captured image. After that, measure the 2D coordinates of the pixel coordinate system calibration point in each calibration point pair in the pixel coordinate system, and obtain the coordinates of each pixel coordinate system calibration point in the pixel coordinate system.

[0045] Afterwards, the rotation vector and translation vector between the camera coordinate system and the world coordinate system are determined based on the second transformation relationship of the coordinates of the four calibration point pairs. Specifically, the point pair coordinates of the 3D coordinates and 2D coordinates of all calibration point pairs, the intrinsic parameter matrix, and the distortion coefficient are used for extrinsic calibration to obtain the rotation vector and translation vector. Then, based on the rotation vector and translation vector, the extrinsic parameter matrix of the fisheye camera is obtained, and the extrinsic parameter matrix is ​​used as the first transformation relationship.

[0046] In the above external parameter calibration process, 4 or more calibration point pairs are required. The world coordinate system calibration points in each calibration point pair are on the image taken by the fisheye camera, and the pixel coordinate system calibration points on the image taken by the fisheye camera correspond to one world coordinate system calibration point. It is necessary to measure the pixel coordinates (x, y) of the pixel coordinate system calibration points in each calibration point pair in the pixel coordinate system, and the coordinates (Xw, Yw, Zw) of the world coordinate system calibration points in each calibration point pair in the world coordinate system. Using the coordinates of multiple calibration point pairs and the intrinsic parameter matrix and distortion coefficient of the fisheye camera in the second transformation relationship, the rotation vector rvec and translation vector tvec of the world coordinate system to the camera coordinate system are obtained, and then the external parameter matrix is ​​obtained to obtain the first transformation relationship.

[0047] For example, when calibrating external parameters, refer to Figure 2, the fisheye camera needs to be fixed. When the plane is set to the ground where the vehicle is located, the calibration can be performed by manually selecting pixel points after taking a picture of the ground. First, the position of the fisheye camera needs to be determined. In smart driving vehicles, fisheye cameras are often used to take surround images and are installed in the front, back, left, and right directions of the vehicle. In some embodiments, the center of the rear axle of the vehicle can be used as the origin of the world coordinate system (or vehicle coordinate system). After determining the position of the fisheye camera, the 3D point with known coordinates on the ground in the world coordinate system is determined. At least 4 world coordinate system calibration points are required, and all 4 world coordinate system calibration points must appear in the picture taken by the fisheye camera. Then use the captured fisheye original image to manually select the 2D coordinates of the 4 pixel coordinate system calibration points in the pixel coordinate system of the image. And the world coordinate system calibration points and pixel coordinate system calibration points in each pixel calibration point pair correspond one to one. Using the 2D point of each calibration point pair in the pixel coordinate system and the 3D point in the world coordinate system, four calibration point pairs are formed. Together with the intrinsic parameter matrix and the distortion coefficient, the 3×1 rotation vector rvec and 3×1 translation vector tvec from the world coordinate system to the camera coordinate system are obtained. The rotation vector rvec can be used to obtain the 3×3 rotation matrix R through the Rodriguez formula shown below.

[0048]

[0049] Where I is the unit matrix, n is the unit vector of the rotation vector, and θ is the modulus of the rotation vector. tvec is the translation vector, which represents the relative displacement between the origin of the world coordinate system and the origin of the camera coordinate system. Let T = tvec, then the external parameter matrix of the fisheye camera is Afterwards, the external parameter matrix can be used as the first conversion relationship.

[0050] It should be understood that, in addition to the above-described method, other methods may be used to determine the first conversion relationship between the camera coordinate system of the fisheye camera and the world coordinate system.

[0051] Exemplarily, when obtaining the first conversion relationship between the camera coordinate system of the fisheye camera and the world coordinate system, the following steps can also be used: first, obtain the position of the optical center of the fisheye camera in the world coordinate system; then, determine the translation vector between the camera coordinate system and the world coordinate system according to the position of the optical center and the deflection angle, and determine the deflection angle of one of the coordinate axes in the rotation vector. Exemplarily, the deflection angle between the Z axis of the camera coordinate system and the Z axis of the world coordinate system can be determined. Among them, when obtaining the position of the optical center of the fisheye camera in the world coordinate system, the coordinate position of the optical center of the fisheye camera in the world coordinate system can be obtained. The optical center of the fisheye camera is the origin of the camera coordinate system. Therefore, the translation vector between the camera coordinate system and the world coordinate system can be determined according to the coordinate position of the optical center of the fisheye camera and the origin position of the world coordinate system. Afterwards, the deflection angles of the other two coordinate axes in the rotation vector can be determined by the directions of the other two coordinate axes defined in the camera coordinates and the world coordinate system to obtain a complete rotation vector. For example, the deflection angle between the X-axis of the camera coordinate system and the X-axis of the world coordinate system, and the deflection angle between the Y-axis of the camera coordinate system and the Y-axis of the world coordinate system can be determined according to the directions of the X-axis and the Y-axis defined in the camera coordinate system and the world coordinate system, respectively, to obtain a complete rotation vector. Then, according to the rotation vector and the translation vector, the extrinsic parameter matrix of the fisheye camera is obtained, and the extrinsic parameter matrix is ​​used as the first conversion relationship. That is, the first conversion relationship finally obtained includes the extrinsic parameter matrix to complete the extrinsic calibration of the fisheye camera.

[0052] In some embodiments, after the installation position of the fisheye camera is fixed, the deflection angle of the optical axis of the fisheye camera relative to the set plane in the world coordinate system can be obtained. Among them, the optical axis of the fisheye camera is a directed line drawn from the center of the lens of the fisheye camera with the optical center of the fisheye camera as the starting point. The deflection angle refers to the angle between the optical axis and the right angle of the set plane. The set plane can be any plane in the world coordinate system. Exemplarily, the set plane can be parallel to the first coordinate axis and the second coordinate axis in the world coordinate system. As an example, the first coordinate axis and the second coordinate axis can be the Xw coordinate axis and the Yw coordinate axis in the world coordinate system, that is, the set plane is a plane perpendicular to the Zw coordinate axis in the world coordinate system. In some embodiments, the set plane can be the XwOwYw plane in the world coordinate system, where Ow is the origin in the world coordinate system. Of course, in other embodiments, when the fisheye camera is applied to a vehicle, the ground where the vehicle is located can be used as the set plane. In some embodiments, the center of the rear axle of the vehicle can be used as the origin of the world coordinate system (or vehicle coordinate system).

[0053] Exemplary, reference Figure 4 and Figure 5 , the optical axis of the fisheye camera can be the Zc coordinate axis in the camera coordinate system. Figure 4 and Figure 5 , the deflection angle can be represented by θ. Exemplarily, the deflection angle θ can be the angle between the Zc coordinate axis in the camera coordinate system and the XwOwYw plane in the world coordinate system. In some embodiments, the deflection angle θ between the optical axis of the fisheye camera and the set plane can be not less than 0° and not greater than 90°, that is, the fisheye camera is set toward the set plane relative to the set plane. Specifically, the deflection angle can be any value between 0° and 90°, such as 0°, 30°, 45°, 60°, 75°, 90°, etc. When the set plane is the ground where the vehicle is located, the fisheye camera is set toward the ground.

[0054] Exemplarily, the following method can be used to obtain the deflection angle of the optical axis of the fisheye camera relative to the set plane in the world coordinate system. Exemplarily, the deflection angle is obtained by solving the first conversion relationship. Specifically, the deflection angle of the optical axis of the fisheye camera relative to the set plane in the world coordinate system can be determined according to the first conversion relationship. Of course, in addition to the methods shown above, other methods can also be used. For example, the deflection angle can also be obtained by pre-calibration based on the installation position of the fisheye camera, and the deflection angle can even be obtained by measurement.

[0055] Next, refer to Figure 1 , according to the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to the plane set in the world coordinate system, the corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system are determined. That is, the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to the plane set in the world coordinate system are used to determine the corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system. Specifically, the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to the plane set in the world coordinate system can be used in a variety of ways.

[0056] Exemplarily, when determining the corresponding coordinates of the target point in the image captured by the fisheye camera in the world coordinate system according to the first conversion relationship, the second conversion relationship and the deflection angle of the optical axis of the fisheye camera relative to the set plane in the world coordinate system, the following method can be adopted: according to the image captured by the fisheye camera, obtain the first coordinate corresponding to the target point in the pixel coordinate system; according to the first coordinate, the second conversion relationship and the deflection angle, determine the second coordinate corresponding to the target point in the camera coordinate system; according to the second coordinate and the first conversion relationship, determine the third coordinate corresponding to the target point in the world coordinate system. That is, in the process of obtaining the second coordinate corresponding to the target point in the camera coordinate system according to the first coordinate corresponding to the target point in the pixel coordinate system, not only the second conversion relationship is used, but also the deflection angle is used, so that the second coordinate corresponding to the target point in the camera coordinate system can be more determined. And there is no need to make the optical center of the fisheye camera and the center of the target detection object in the same horizontal plane, support different installation angles of the fisheye camera, reduce installation requirements and installation difficulty, and have more usage scenarios.

[0057] When determining the second coordinate corresponding to the target point in the camera coordinate system according to the first coordinate, the second conversion relationship and the deflection angle, a variety of methods can be used. Exemplarily, the following method can be used. First, obtain the vertical distance of the target point from the set plane in the world coordinate system; then, determine the third coordinate corresponding to the target point in the camera coordinate system according to the first coordinate, the second conversion relationship, the deflection angle θ and the vertical distance.

[0058] That is, in the image taken by the fisheye camera with a fixed position, a target point can be selected, and the first coordinate corresponding to the target point in the pixel coordinate system can be obtained. Afterwards, the vertical distance of the target point from the set plane in the world coordinate system is also obtained. Since in the aforementioned calibration process, the first conversion relationship includes the vertical distance from the origin of the camera coordinate system to the set plane, the distance H between the target point and the origin of the camera coordinate system in the direction perpendicular to the set plane can be determined based on the vertical distance from the origin of the camera coordinate system to the set plane and the vertical distance from the target point to the set plane. Afterwards, the second coordinate corresponding to the target point in the camera coordinate system can be determined based on the first coordinate, the second conversion relationship, the deflection angle θ and the vertical distance H. In some embodiments, when the target point is located on the set plane, the vertical distance of the target point from the set plane is equal to the vertical distance from the origin of the camera coordinate system to the set plane. Exemplarily, the set plane may be the XwOwYw plane in the world coordinate system, and the target point may be a point on the XwOwYw plane in the world coordinate system, so as to accurately obtain the target point whose coordinates are the first coordinates in the pixel coordinate system and the corresponding third coordinates in the world coordinate system, so as to obtain parameters such as the distance between different points in the world coordinate system.

[0059] It needs to be explained that the reference Figure 3 In the embodiment of the present application, when determining the second coordinate Pc corresponding to the target point in the camera coordinate system according to the deflection angle and the second conversion relationship, the deflection angle θ and the vertical distance H are mainly used to solve the Zc in the second coordinate Pc. In the prior art, because the optical axis of the fisheye camera is parallel to the ground, the Zc of all points is a fixed value. Therefore, compared with the prior art, the present application does not need to make the optical center of the fisheye camera and the center of the target detection object in the same horizontal plane, supports different installation angles of the fisheye camera, reduces installation requirements and installation difficulty, and has more usage scenarios.

[0060] Exemplary, reference Figure 2 , the target detection method may also include: using a target detection algorithm to detect the target object on the image, and determining the target point according to the target object. That is, when obtaining the target point from the image taken by the fisheye camera, the present application also discloses a method of selecting the target point in some embodiments, specifically using a target detection algorithm to detect the target object on the image, and determining the target point according to the target object on the image. There are many ways to detect, and several methods are introduced as examples below.

[0061] Exemplarily, when a target detection algorithm is used to detect a target object on an image and a target point is determined based on the target object, the following steps may be used. First, a detection frame of the target detection algorithm is used to frame the target object on the image; then, the target point of the target object is determined based on the detection frame. Specifically, the detection frame of the target detection algorithm is first used to frame the target object on the image. The target object may be an object of regular shape, such as a rectangular, circular, or other regular shape. In other embodiments, the target object may also be an object of irregular shape. For example, the target object may be an object such as, but not limited to, a person, a telephone pole, a vehicle, an ice cream cone, etc., and the target object is not required to be a cube. The shape of the detection frame may be, for example, but not limited to, a rectangular frame, etc. In some embodiments, when the number of target object categories increases, other categories may be added through training to optimize the target detection algorithm, and the number of categories may be increased through model training to improve the accuracy, and operations such as ranging may be performed on the target of interest.

[0062] After the target object is framed, a target point representing the position of the target object can be selected according to the detection frame, and the position of the target object can be represented by the position of the target point. When making specific selections, a variety of methods can be used. Exemplarily, when determining the target point of the target object according to the detection frame, the midpoint of the lower edge of the detection frame can be used as the target point. It should be understood that the method of determining the target point of the target object according to the detection frame is not limited to the method shown above, and other methods can also be used. For example, the endpoint of the lower edge of the detection frame can also be selected as the target point, or the center point of the detection frame can be selected as the target point.

[0063] Exemplary, reference Figure 2 , the target detection method may further include: determining the distance between the target point and the set point according to the third coordinate and the set coordinate of the set point in the world coordinate system. In some embodiments, the set point may be a pre-calibrated reference point. For example, the set point is the origin of the world coordinate system or the optical center of the fisheye camera, and its coordinates in the world coordinate system are known set coordinates. In other embodiments, the set point may be another target point picked up from the image taken by the fisheye camera, and the above-mentioned conversion relationship is also used to obtain the coordinate information of the target point in the world coordinate system.

[0064] Exemplarily, when the distance between the target point and the set point is determined according to the third coordinate and the set coordinate of the set point in the world coordinate system, the distance may be the distance of a direct line between the target point and the set point. The distance between the target point and the set point may be determined according to the third coordinate and the set coordinate of the set point in the world coordinate system, and the specific calculation method may be a method of calculating the distance between two points in the same coordinate system.

[0065] In other embodiments, when determining the distance between the target point and the set point according to the third coordinate and the set coordinate of the set point in the world coordinate system, the first projection point and the second projection point of the target point and the set point, respectively, which are vertically projected on the set plane, can be determined according to the third coordinate and the set coordinate, and the distance between the first projection point and the second projection point can be determined. That is, first determine the first projection point of the vertical projection of the target point on the set plane, and the second projection point of the vertical projection of the set point on the set plane. Afterwards, calculate the distance between the first projection point and the second projection point according to the third coordinate and the set coordinate. When the set plane is the plane where the vehicle is located, the distance between the first projection point and the second projection point represents the horizontal distance between the target point as the grounding point and the set point, thereby facilitating the calculation of the distance between the target object and the vehicle.

[0066] For example, after the calibration is completed, the fisheye camera can be used to measure the target distance, which requires that the fisheye camera's posture, that is, the installation position and angle must be consistent with the external parameter calibration. The following takes the setting plane as the ground where the vehicle is located, the target object as the object on the ground, the target point as the target ground point where the target object contacts the ground, the setting plane as the distance from the optical center of the fisheye camera as H, and the deflection angle between the optical axis of the fisheye camera and the setting plane as θ as an example to introduce the principle of the relevant conversion of the coordinates of the target point between the pixel coordinate system, the camera coordinate system and the world coordinate system.

[0067] The conversion formula from a 3D point in the world coordinate system to a 2D point in the pixel coordinate system is:

[0068]

[0069] And the conversion formula from the point Pw (Xw, Yw, Zw) in the world coordinate system to the point Pc (Xc, Yc, Zc) in the camera coordinate system is,

[0070]

[0071] First, the target detection algorithm is used to process the distorted original image taken by the fisheye camera, and a detection frame is output for the target object of interest. The lower midpoint of the detection frame is used as the ground point on the image to obtain a 2D point in the pixel coordinate system, that is, Figure 3 The point P(x,y) is shown.

[0072] From formulas (1) and (2), we can get:

[0073]

[0074] Right now,

[0075]

[0076] From formula (3), we can see that to transform the point P(x, y) in the pixel coordinate system to the point Pc(Xc, Yc, Zc) in the camera coordinate system, we need to know the projection Zc of the point Pc on the Z axis of the camera coordinate system and the camera's intrinsic parameter matrix K. The 2D point P(x, y) of the target object in the pixel coordinate system and the intrinsic parameter matrix K of the fisheye camera are known. Let but In (5), the left value of the equal sign is known. Now we need to solve Pc(Xc,Yc,Zc).

[0077] According to formula (5),

[0078]

[0079] Transformation is available,

[0080]

[0081] like Figure 4 As shown, the solution of Zc is discussed in two cases:

[0082] ① If the 3D point is on the left side of the intersection of the camera optical axis and the Yw axis of the world coordinate system, there is ΔAOwC~ΔBPcC. According to the principle of similar triangles, we have

[0083]

[0084] Given the camera height H and camera installation angle θ, we have:

[0085]

[0086] Substituting formula (9) into formula (8), we can obtain:

[0087]

[0088] After simplification, substitute formula (7) into the equation to obtain:

[0089]

[0090] ②If the 3D point is on the right side of the intersection of the camera optical axis and the Yw axis of the world coordinate system, the area S of ΔCOcPc′ is:

[0091]

[0092] Therefore,

[0093] |OcC||P'D|=H|CP′| (11)

[0094] P'(Xc', Yc', Zc') is on the negative half axis of the camera's Yc axis, so Yc' is a negative number. Substituting the camera height H and angle θ into formula (11), we can get:

[0095]

[0096] After simplification, substitute formula (7) into the equation to obtain:

[0097]

[0098] In summary, formula (10) is applicable to all points in the camera coordinate system.

[0099] The point (Xc, Yc, Zc) in the camera coordinate system is transferred to the world coordinate system with Ow as the origin, such as Figure 4 As shown, according to formula (2), we have

[0100]

[0101] External parameter matrix It has been obtained through calibration. Substituting it into the above formula, we can get the position of the grounded 3D point of the target object in the world coordinate system as (Xw, Yw, Zw), and the distance L from the world coordinate system is,

[0102]

[0103] In the distance measurement method shown above, the target point is directly selected from the image taken by the fisheye camera, and contour extraction is not required, thus eliminating the step of contour extraction. The distance from the target object to the set point is calculated using the midpoint of the lower edge of the detection frame as the target point. Under the premise of ensuring the accuracy of distance measurement, the algorithm steps are simpler. That is, by directly using the original fisheye image for target detection and distance measurement, the stretching and cropping of the object by the dedistortion image is avoided, and the image distortion correction steps are reduced, which can reduce the error effect of the internal reference on the distance measurement, while ensuring the detection range and the number of target objects detected. Moreover, the center of the target object may not be kept in the same horizontal plane as the optical center of the fisheye camera, which can solve the distance measurement problem when the center of the target object is at different positions on the optical axis of the camera, and the distance is calculated using the target ground point as the target point, which has a wider range of application.

[0104] It can be seen from the distance measurement part shown above that first, the image is captured by a fisheye camera, and the distorted original image is sent to the target detection algorithm model for target detection. The target detection algorithm model of this part can be replaced with any suitable target detection model according to the hardware platform requirements or system requirements. After target detection, a 2D detection frame is obtained, and the midpoint of the lower frame of the 2D detection frame is used as the first coordinate of the 2D grounding point (target point) of the target object in the image. Through the intrinsic matrix and the distortion coefficient, the 2D target point is converted to the camera coordinate system, and then converted to the world coordinate system through the extrinsic matrix to obtain the 3D grounding point of the target object in the world coordinate system. The 3D grounding point is the third coordinate of the target point in the world coordinate system. Finally, the distance from the target object to the origin of the world coordinate system or the optical center of the fisheye camera is calculated based on the coordinate information of the 3D grounding point.

[0105] In the various embodiments shown above, the first conversion relationship between the camera coordinate system of the fisheye camera and the world coordinate system, and the second conversion relationship between the camera coordinate system of the fisheye camera and the pixel coordinate system are first obtained; then the corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system are determined according to the first conversion relationship, the second conversion relationship, and the deflection angle of the optical axis of the fisheye camera relative to the set plane in the world coordinate system. Compared with the installation requirement of the prior art that the optical center of the fisheye camera and the center of the target detection object need to be in the same horizontal plane, the present application determines the corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system according to the first conversion relationship, the second conversion relationship, and the deflection angle of the optical axis of the fisheye camera relative to the set plane in the world coordinate system after completing the internal and external parameter calibration. It is not necessary to make the optical center of the fisheye camera and the center of the target detection object in the same horizontal plane, support different installation angles of the fisheye camera, reduce installation requirements and installation difficulty, use more scenarios, and the coordinates of the determined target point in the world coordinate system are closer to the position of the real scene.

[0106] Compared with the related art which is only applicable to the distance measurement between the fisheye camera and the plane chessboard, and the size of the chessboard is known, and cannot be applied to the distance measurement of target objects of unknown size in the intelligent driving vehicle, the distance measurement method provided in some embodiments of the present application can be applied to the distance measurement of objects of different categories, and the size of the object does not need to be measured in advance, which is more practical and has a wider scope of application.

[0107] Compared with the related art, it is necessary to ensure that the optical center of the fisheye camera lens and the center of the object are on the same horizontal plane, and the framed image needs to be contour recognized and feature points extracted. The applicable distance measurement conditions are more stringent, the height of the target object must be a fixed value, and the algorithm steps are more. The distance measurement method provided in some embodiments of the present application can be applied to common road objects, has no specific requirements for height, and directly uses the midpoint of the lower edge of the output frame of the target detection algorithm as the grounding point of the object for distance measurement, which ensures the accuracy of distance measurement and makes the algorithm simpler.

[0108] Compared with the use of fisheye dedistortion images for target ranging in related technologies, the edges of the dedistortion images will stretch the target object, affecting the detection effect. If the image is scaled and sheared, the detection range and target object will be reduced, affecting the detection effect. In some embodiments of the present application, the use of fisheye original images for target detection and ranging can reduce the error impact of internal parameters while ensuring the detection range and the number of detected target objects.

[0109] Compared with the prior art, the above embodiments of the present application have the following differences:

[0110] 1. The prior art requires that the optical axis of the fisheye camera must be level with the ground, but in reality, the fisheye camera is installed at a certain angle. In some embodiments of the present application, the different purposes and installation angles of the fisheye camera are taken into consideration, and the fisheye camera can be used for non-horizontal cameras, that is, when the optical axis of the fisheye camera is not parallel to the horizontal plane.

[0111] 2. The prior art requires that the center of the target object and the optical center of the camera be in the same horizontal plane. However, in actual applications, the object may be located at different positions along the optical axis of the camera, and it is difficult to ensure that the center is always on the optical axis of the camera. In some embodiments of the present application, the center of the target object may not be kept in the same horizontal plane as the optical center of the fisheye lens, and grounded target objects are applicable;

[0112] 3. The target object applicable to the prior art is required to be a cube, which has a narrow scope of application. In some embodiments of the present application, the target object applicable to the present invention may be a non-cubic ground target such as a person, a vehicle, or a chair, and other target categories may be added through training;

[0113] 4. The prior art uses ResNet for target detection, performs image processing on the obtained frame diagram to obtain the target contour diagram, performs contour tracking, obtains the feature points of the target object based on the feature points of the contour, and solves the distance based on the feature points. This method is too complicated. In some embodiments of the present application, a target detection algorithm is used to output the detection frame of the target object in real time, and the distance from the midpoint of the ground edge of the detection frame to the camera is measured, eliminating the step of solving the feature points;

[0114] 5. The prior art uses fisheye dedistorted images for target ranging. The edges of the dedistorted images will stretch the target object, affecting the detection effect. After the image is scaled and sheared, the detection range and target object will be reduced, affecting the detection effect. In some embodiments of the present application, the original fisheye image is directly used for target detection and ranging, which can reduce the error effect of the internal reference on the ranging, while ensuring the detection range and the number of detected target objects.

[0115] Furthermore, an embodiment of the present application also provides a target detection device based on a fisheye camera, which includes: a storage medium and a processor, the storage medium stores a computer program run by the processor, and when the computer program is run by the processor, the processor executes any one of the above-mentioned target detection methods based on the fisheye camera.

[0116] Figure 6 FIG. 1 is a schematic block diagram of a target detection device 100 based on a fisheye camera according to an embodiment of the present application. Figure 6 As shown, the target detection device 100 based on a fisheye camera according to an embodiment of the present application may include a storage medium 110 and a processor 120. The storage medium 110 stores a computer program executed by the processor 120. When the computer program is executed by the processor 120, the processor 120 executes the target detection method based on a fisheye camera according to the embodiment of the present application described above. Those skilled in the art can understand the specific operation of the target detection device 100 based on a fisheye camera according to the embodiment of the present application in combination with the above content, and for the sake of brevity, it will not be repeated here.

[0117] The storage medium 110 may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0118] In addition, the embodiment of the present application further provides a vehicle, which includes: a vehicle body, a fisheye camera arranged on the vehicle body, and any of the above-mentioned target detection devices based on the fisheye camera. The vehicle can be specifically such as but not limited to an electric vehicle, a hybrid vehicle, a fuel vehicle, etc., that is, any vehicle equipped with a fisheye camera can be used as a vehicle in the embodiment of the present application. The fisheye camera can be installed around the body of the vehicle to detect target objects around the body of the vehicle, and then measure the distance of the detected target objects. The measurement method can adopt the method shown in the above-mentioned embodiment.

[0119] The present invention has been described by the above embodiments, but it should be understood that the above embodiments are only for the purpose of example and description, and are not intended to limit the present invention to the scope of the described embodiments. In addition, it can be understood by those skilled in the art that the present invention is not limited to the above embodiments, and more variations and modifications can be made according to the teachings of the present invention, and these variations and modifications all fall within the scope of the protection claimed by the present invention. The protection scope of the present invention is defined by the attached claims and their equivalents.

Claims

1. A target detection method based on a fisheye camera, characterized in that: include: Acquire a first conversion relationship between a camera coordinate system of the fisheye camera and a world coordinate system, and a second conversion relationship between the camera coordinate system of the fisheye camera and a pixel coordinate system; The corresponding coordinates of the target point in the image taken by the fisheye camera in the world coordinate system are determined according to the first conversion relationship, the second conversion relationship, and the deflection angle of the optical axis of the fisheye camera relative to a set plane in the world coordinate system.

2. The target detection method according to claim 1, characterized in that: Also includes: The deflection angle is pre-calibrated according to the installation position of the fisheye camera; or, The deflection angle is calculated according to the first conversion relationship.

3. The target detection method according to claim 1, characterized in that: The deflection angle between the optical axis of the fisheye camera and the setting plane is not less than 0° and not more than 90°.

4. The target detection method according to claim 1, characterized in that: The determining, according to the first conversion relationship, the second conversion relationship, and the deflection angle of the optical axis of the fisheye camera relative to a set plane in the world coordinate system, corresponding coordinates of a target point in the image captured by the fisheye camera in the world coordinate system includes: According to the image taken by the fisheye camera, obtaining a first coordinate corresponding to the target point in the pixel coordinate system; Determine a second coordinate corresponding to the target point in the camera coordinate system according to the first coordinate, the second conversion relationship and the deflection angle; A third coordinate corresponding to the target point in the world coordinate system is determined according to the second coordinate and the first conversion relationship.

5. The target detection algorithm according to claim 4, characterized in that: The determining, according to the first coordinate, the second conversion relationship and the deflection angle, a second coordinate corresponding to the target point in the camera coordinate system includes: Obtaining a vertical distance between the target point and the set plane in the world coordinate system; The second coordinate corresponding to the target point in the camera coordinate system is determined according to the first coordinate, the second conversion relationship, the deflection angle and the vertical distance.

6. The target detection method according to claim 4, characterized in that: Also includes: A target detection algorithm is used to detect the target object on the image, and the target point is determined according to the target object.

7. The target detection method according to claim 6, characterized in that: The adopting a target detection algorithm to detect the target object on the image, and determining the target point according to the target object, comprises: Using the detection frame of the target detection algorithm to frame the target object on the image; The target point of the target object is determined according to the detection frame.

8. The target detection method according to claim 7, characterized in that: Determining the target point of the target object according to the detection frame includes: The midpoint of the lower edge of the detection frame is used as the target point.

9. The target detection method according to claim 8, characterized in that: Also includes: The distance between the target point and the set point is determined according to the third coordinate and the set coordinate of the set point in the world coordinate system.

10. The target detection method according to claim 9, characterized in that: The step of determining the distance between the target point and the set point according to the third coordinate and the set coordinate of the set point in the world coordinate system comprises: According to the third coordinate and the set coordinate, determine the first projection point and the second projection point of the target point and the set point vertically projected on the set plane respectively, and determine the distance between the first projection point and the second projection point.

11. The target detection method according to claim 9, characterized in that: The set point is the origin of the world coordinate system or the optical center of the fisheye camera.

12. The target detection method according to claim 1, characterized in that: The setting plane is parallel to both the first coordinate axis and the second coordinate axis in the world coordinate system.

13. The target detection method according to claim 1, characterized in that: The obtaining of a first conversion relationship between a camera coordinate system of the fisheye camera and a world coordinate system comprises: Acquire coordinates of at least four calibration point pairs, each calibration point pair comprising: a pixel coordinate system calibration point, and a world coordinate system calibration point corresponding to the pixel coordinate system calibration point; wherein the coordinates of the pixel coordinate system calibration point are the coordinates of the pixel coordinate system calibration point in the pixel coordinate system, the coordinates of the world coordinate system calibration point are the coordinates of the world coordinate system calibration point in the world coordinate system, and all the world coordinate system calibration points are located in the set plane; Determine a rotation vector and a translation vector between the camera coordinate system and the world coordinate system according to the coordinates of at least four calibration point pairs and the second transformation relationship; An extrinsic parameter matrix of the fisheye camera is obtained according to the rotation vector and the translation vector, and the extrinsic parameter matrix is ​​used as the first conversion relationship.

14. The target detection method according to claim 1, characterized in that: The second conversion relationship includes the intrinsic parameter matrix and distortion coefficient of the fisheye camera.

15. A target detection device based on a fisheye camera, characterized in that: include: A storage medium and a processor, wherein the storage medium stores a computer program executed by the processor, and when the computer program is executed by the processor, the processor executes the target detection method based on a fisheye camera according to any one of claims 1 to 14.

16. A vehicle, characterized in that: include: Vehicle body; A fisheye camera disposed on the vehicle body; The target detection device based on a fisheye camera as described in claim 15 is arranged on the vehicle body.

Images