Parameter determination method and device of fisheye camera, electronic equipment and storage medium
Through the adaptive calculation of the target center point of the fisheye camera, the distortion matrix and detection model are used to solve the problem of inaccurate calculation of the target center point caused by the distortion of the fisheye camera, and a more accurate and accurate center point selection is achieved.
Patent Information
- Application Number
- CN202510108805.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-23
- Publication Date
- 2025-05-30
AI Technical Summary
Due to the large distortion of the fisheye camera in autonomous driving, the calculation of the target center point is inaccurate, especially when the distortion is different in different areas, there are different degrees of deviations in the center point of the BBOX.
By referring to the distortion degree of each point in the fisheye camera distortion matrix, the target center point is adaptively calculated, and a dedistortion processing and a pre-trained detection model are used to detect the target bounding box, calculate the compensation value, and compensate the center point parameters.
Improve the accuracy of the target center point, reduce position errors, and achieve a more accurate center point selection.
Smart Images
Figure CN120070272A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and particularly to a method and device for determining parameters of a fish-eye camera, an electronic device, and a storage medium. Background Art
[0002] In the road-end autonomous driving solution, in order to monitor motor vehicles, non-motor vehicles, and pedestrians without dead angles, a combination of a bullet camera and a fish-eye camera is usually adopted. Among them, the bullet camera is used to monitor the distance, and the fish-eye camera is used to monitor the blind area under the pole.
[0003] Since the distortion at the edge of the fish-eye camera is much greater than that under the pole, if the center of the target bounding box (BBOX) of the target is directly calculated as the target center point, deformation will occur. Therefore, the calculation of the center point at the edge of the fish-eye camera will be very inaccurate. Summary of the Invention
[0004] Embodiments of this application provide a method and device for determining parameters of a fish-eye camera, an electronic device, and a storage medium. By referring to the distortion degree of each point in the distortion matrix of the fish-eye camera, the target center point is adaptively calculated, making the target center point more accurate and the position error smaller.
[0005] Embodiments of this application adopt the following technical solutions:
[0006] In a first aspect, embodiments of this application provide a method for determining parameters of a fish-eye camera, where the method includes:
[0007] Undistort the fish-eye camera according to the distortion matrix of the fish-eye camera to obtain an undistorted fish-eye camera image;
[0008] Detect the target bounding box in the undistorted fish-eye camera image according to a pre-trained detection model;
[0009] Determine a first compensation value according to the distance between any target bounding box and the image center;
[0010] Determine a second compensation value according to the distortion matrix of the fish-eye camera;
[0011] Compensate the first center point parameter of the current target bounding box through the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box.
[0012] In some embodiments, determining a first compensation value according to the distance between any target bounding box and the image center includes:
[0013] Initialize the image center point and the center point in the target bounding box;
[0014] Calculate the offset distance of the center point of the target bounding box relative to the center point of the image based on the center point of the image and the center point in the target bounding box;
[0015] Based on the offset distance of the center point of the target bounding box relative to the center point of the image, the width and height of any target bounding box, calculate the offset of the center point of the target bounding box and use it as the first compensation value.
[0016] In some embodiments, the determining the second compensation value according to the distortion matrix of the fisheye camera includes:
[0017] Calculate the distance from the target original coordinate point of the original image of the fisheye camera to the center point coordinate of the original image;
[0018] According to the distortion matrix of the fisheye camera, calculate the pixel coordinates after image undistortion;
[0019] According to the pixel coordinates after image undistortion and the target original coordinate point of the original image, obtain the distortion rate and use it as the second compensation value.
[0020] In some embodiments, the distortion matrix of the fisheye camera includes:
[0021] Obtain the target pixel coordinates in the original image of the fisheye camera;
[0022] Determine the distance from the target pixel coordinates to the center point of the image;
[0023] Determine the radial distortion coefficient and tangential distortion coefficient according to the internal parameters of the fisheye camera;
[0024] According to the target pixel coordinates in the original image of the fisheye camera, the distance from the target pixel coordinates to the center point of the image, the radial distortion coefficient and the tangential distortion coefficient, calculate the corrected pixel coordinates and use them as the distortion matrix of the fisheye camera.
[0025] In some embodiments, detecting the target bounding box in the undistorted fisheye camera image according to a pre-trained detection model includes:
[0026] Detect a set of target bounding boxes of the undistorted fisheye camera image according to a pre-trained detection model, and the set of target bounding boxes includes at least one of the following: the category of the target in the bounding box, the confidence score of the bounding box, the width and height of the bounding box, and the corner position coordinates of the bounding box.
[0027] In some embodiments, undistorting the fisheye camera according to the distortion matrix of the fisheye camera to obtain an undistorted fisheye camera image includes:
[0028] Map the pixels in the original image in the fisheye camera to the corrected pixel positions according to the distortion matrix of the fisheye camera, and obtain the undistorted fisheye camera image.
[0029] In a second aspect, an embodiment of the present application further provides an image processing method for a fisheye camera, which includes the method for determining the parameters of the fisheye camera described in the first aspect.
[0030] In a third aspect, an embodiment of the present application further provides a parameter determination device for a fisheye camera, where the device includes:
[0031] An undistortion module, configured to undistort the fisheye camera according to the distortion matrix of the fisheye camera, and obtain an undistorted fisheye camera image;
[0032] A bounding box detection module, configured to detect a target bounding box in the undistorted fisheye camera image according to a pre-trained detection model;
[0033] A first compensation determination module, configured to determine a first compensation value according to the distance between any target bounding box and the image center;
[0034] A second compensation determination module, configured to determine a second compensation value according to the distortion matrix of the fisheye camera;
[0035] A compensation module, configured to compensate the first center point parameter of the current target bounding box through the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box.
[0036] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, and the executable instructions, when executed, cause the processor to execute the above method.
[0037] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute the above method.
[0038] The above at least one technical solution adopted in the embodiments of the present application can achieve the following beneficial effects: The fisheye camera is undistorted according to the distortion matrix of the fisheye camera to obtain an undistorted fisheye camera image, and then the target bounding box in the undistorted fisheye camera image is detected according to a pre-trained detection model. Further, a first compensation value is determined according to the distance between any target bounding box and the image center, and a second compensation value is determined according to the distortion matrix of the fisheye camera. Finally, the first center point parameter of the current target bounding box is compensated by the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box. In the above method, position linear compensation and position distortion compensation are combined to select a more accurate point for the center point of the image after fisheye undistortion. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:
[0040] Figure 1 is a schematic diagram of a fisheye camera image before undistortion in an embodiment of the present application;
[0041] Figure 2 is a schematic diagram of a fisheye camera image after undistortion in an embodiment of the present application;
[0042] Figure 3 is a schematic flowchart of a method for determining parameters of a fisheye camera in an embodiment of the present application;
[0043] Figure 4 is a schematic structural diagram of a device for determining parameters of a fisheye camera in an embodiment of the present application;
[0044] Figure 5 is a schematic structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0045] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.
[0046] Due to the large distortion of the fisheye camera in the related art, the common practice is to first remove the fisheye distortion, then perform object detection, and after obtaining the object center point, map the center point to the UTM coordinates to obtain the longitude and latitude of the object. However, the method for calculating the center point of the fisheye object in the related art is too simple and crude. It directly calculates the center of the BBOX of the object as the object center point. However, since the distortion at the edge of the fisheye is much greater than that under the pole, the vehicle will be deformed, so the calculation of the center point at the edge of the fisheye will be very inaccurate. Further, due to the different distortions of the fisheye camera in different regions, there will also be different degrees of deviation in the BBOX center point.
[0047] In view of the above deficiencies, an adaptive calculation method is proposed in the embodiments of the present application. It will adaptively calculate the center point by referring to the distortion degree of each point in the distortion matrix. The center point calculated in this way is more accurate and the position error is also smaller.
[0048] The following will describe in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.
[0049] The embodiments of the present application provide a method for determining the parameters of a fisheye camera. As Figure 3 shown, a schematic flow diagram of the method for determining the parameters of the fisheye camera in the embodiments of the present application is provided. The method at least includes the following steps S310 to S350:
[0050] Step S310, undistort the fisheye camera according to the distortion matrix of the fisheye camera to obtain an undistorted fisheye camera image.
[0051] In the road-end autonomous driving solution, the fisheye camera is mainly used to monitor the blind area under the pole. However, due to the image distortion of the fisheye camera and the different distortions of the fisheye camera in different regions, if the simple method of using the center point of the target image as the target center point is used, there will be a large deviation in the position of the target center point.
[0052] The distortion matrix of the fisheye camera can be obtained by relevant calculations, and no specific limitation is made in the embodiments of the present application. After undistorting the fisheye camera according to the obtained distortion matrix of the fisheye camera, an undistorted fisheye camera image is obtained. As Figure 1 shown, it is a schematic diagram before undistorting the fisheye camera. As Figure 2 shown, it is a schematic diagram after undistorting the fisheye camera.
[0053] Step S320, detect the object bounding box in the undistorted fisheye camera image according to the pre-trained detection model.
[0054] The pre-trained detection model can use a target detection model known in the related art or a model for a specific scenario, which is not specifically limited in the embodiments of the present application, as long as it can meet the functions of target detection.
[0055] Detect the target bounding box BBOX in the undistorted fish-eye camera image through the pre-trained detection model. Exemplarily, the results output by the model include multiple target bounding boxes, and a set of target bounding boxes obtained from the multiple target bounding boxes. Each target bounding box includes, but is not limited to, the width and height of the display bounding box, and the target categories in the bounding box, such as motor vehicles, pedestrians, or non-motor vehicles, or cars, large trucks, buses. In addition, the target bounding box also includes a confidence score. It can be understood that the confidence score is used as a basis or weight coefficient for target bounding box selection.
[0056] Step S330: Determine a first compensation value according to the distance between any target bounding box and the image center.
[0057] The "first compensation value", as a linear compensation value, can be used to make a more reasonable compensation for the calculation of the center point later. It should be noted that the process of determining the "first compensation value" has nothing to do with whether the image is undistorted or whether the image has been detected by a pre-trained detection model. That is to say, the process of determining the "first compensation value" can be an independent process or can be pre-determined.
[0058] Exemplarily, first calculate the width and height of the target bounding box BBOX according to the distortion matrix of the fish-eye camera, then calculate the offset of the center point of the target box, and finally obtain the linear compensation adjustment values in the x and y dimensions.
[0059] Step S340: Determine a second compensation value according to the distortion matrix of the fish-eye camera.
[0060] The "second compensation value", as a distortion rate, can be used to make a more reasonable compensation for the calculation of the center point later. It should be noted that the process of determining the "second compensation value" has nothing to do with whether the image is undistorted or whether the image has been detected by a pre-trained detection model. That is to say, the process of determining the "second compensation value" can be an independent process or can be pre-determined.
[0061] Exemplarily, first calculate the distance from the original point (any position) in any target bounding box to the center point, and then calculate the pixel coordinates after undistortion. Then, according to the original point and the pixel coordinates after undistortion, finally, the distortion rates in the x dimension and the y dimension can be calculated.
[0062] Step S350: Compensate the first center point parameter of the current target bounding box through the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box.
[0063] Compensate the first center point parameter of the current target bounding box with the first compensation value and the second compensation value respectively, and finally obtain the second center point parameter of the current target bounding box. It can be understood that the "first center point parameter" refers to the center point position coordinates determined initially, and the "second center point parameter" refers to the center point position coordinates obtained after being compensated by the compensation value. At this time, the obtained center point position coordinates are more accurate and have smaller errors compared with the initially obtained center point position coordinates.
[0064] Through the above method, calculate the distortion matrix of the fisheye camera, and thus obtain the distortion degree of each pixel point according to the distortion matrix. Calculate the distance between the target and the center point of the image to perform linear compensation on the center of the BBOX; finally, calculate the position of the BBOX relative to the center of the image, and based on the center of the BBOX, perform distortion degree compensation and linear distance compensation on the center of the BBOX, so as to accurately give the center point of the target.
[0065] Different from the method in the related art that directly calculates the center of the BBOX of the target as the center point of the target, through the above method, the selection of the center point after fisheye undistortion that combines position linear compensation and position distortion compensation is realized, and the above method is more adaptable.
[0066] In an embodiment of the present application, determining the first compensation value according to the distance between any target bounding box and the center of the image includes: initializing the center point of the image and the center point in the target bounding box; calculating the offset distance of the center point of the target bounding box relative to the center point of the image according to the center point of the image and the center point in the target bounding box; based on the offset distance of the center point of the target bounding box relative to the center point of the image, the width and height of any target bounding box, calculate the offset of the center point of the target bounding box and use it as the first compensation value.
[0067] Calculate the distance between the target bounding box BBOX and the center of the image:
[0068] (1) Initialize the center point of the image: fisheye x =1024 / 2,fisheye y =1024 / 2, where 1024 is the length and width of the image.
[0069] (2) Initialize the center point of the BBOX: bbox x =w i / 2.0,bbox y =h i / 2.0.
[0070] (3) Calculate the offset of the center point of the target box relative to the center point of the image:
[0071] dx=bboxx -fisheye x ;
[0072] dy = bbox y -fisheye y ;
[0073] (4) Calculate the linear compensation value based on distance:
[0074] First, calculate the width and height of the target BBOX: bbox w = w i ; bbox h = h i .
[0075] Then, calculate the offset of the center point of the target box: sigma1 = 1.2, sigma2 = 3.0.
[0076] delta x = (bbox w / 2.0) * pow((abs(dx) / img center.x ), sigma1);
[0077] delta y = (bbox h / 4.0) * pow((abs(dy) / img center.y ), sigma2).
[0078] ·
[0079] ·
[0080] Among them, delta x , delta y are the linear compensation adjustment values in the x and y dimensions, that is, the first compensation value.
[0081] In an embodiment of the present application, the determining the second compensation value according to the distortion matrix of the fisheye camera includes: calculating the distance from the target original coordinate point of the original image of the fisheye camera to the center point coordinate of the original image; calculating the pixel coordinate after image undistortion according to the distortion matrix of the fisheye camera; obtaining the distortion rate based on the pixel coordinate after image undistortion and the target original coordinate point of the original image and using it as the second compensation value.
[0082] Calculate the distance r from the original point to the midpoint: (c x , c y ) is the coordinate of the fisheye center point.
[0083] According to the distortion matrix, calculate the undistorted pixel coordinates (x corrected , y corrected ). Calculate the distortion rate: Assume the original point is (x, y), and the corrected point is (x corrected , y corrected) . Then the distortion rate in the x dimension and the distortion rate in the y dimension are:
[0084]
[0085] Obtain the distortion rate x and the distortion rate y, which are represented by distortion x , distortion y respectively, and can be used for more reasonable compensation in the subsequent calculation of the center point.
[0086] In an embodiment of the present application, the distortion matrix of the fisheye camera includes: obtaining the target pixel coordinates in the original image of the fisheye camera; determining the distance from the target pixel coordinates to the image center point; determining the radial distortion coefficient and the tangential distortion coefficient according to the internal parameters of the fisheye camera; and calculating the corrected pixel coordinates based on the target pixel coordinates in the original image of the fisheye camera, the distance from the target pixel coordinates to the image center point, the radial distortion coefficient, and the tangential distortion coefficient, and using them as the distortion matrix of the fisheye camera.
[0087] For the calculation of the distortion matrix of the fisheye camera, in the embodiment of the present application, a new calculation model based on the combination of radial distortion and tangential distortion is provided. The formula for calculating the distortion matrix of the fisheye camera is as follows:
[0088] x corrected = x(1 + k 1 r 2 + k 2 r 4 + k 3 r 6 ) + 2p 1 xy + p 2 (r 2 + 2x 2 )
[0089] y corrected = y(1 + k 1 r 2 + k 2 r 4 + k 3 r 6 ) + p 1 (r 2 + 2y 2 ) + 2p 2 xy
[0090] (x, y) are the pixel coordinates in the original image.
[0091] (x corrected ,y corrected ) are the corrected pixel coordinates.
[0092] r is the distance from the point (x, y) to the center point of the image.
[0093] k 1 ,k 2 ,k 3 are the radial distortion coefficients and can be obtained through the internal parameters.
[0094] p 1 ,p 2 are the tangential distortion coefficients and can also be obtained through the camera internal parameters.
[0095] It can be understood that the distortion matrix of the fisheye camera is usually a transformation matrix that maps the original pixel coordinates to the corrected pixel coordinates. According to the above matrix model, the distortion matrix of the fisheye camera can be calculated. Rearranging the above formula into matrix form, a transformation matrix can be obtained.
[0096] In an embodiment of the present application, detecting the target bounding box in the undistorted fisheye camera image according to a pre-trained detection model includes: detecting a set of target bounding boxes of the undistorted fisheye camera image according to the pre-trained detection model, and the set of target bounding boxes includes at least one of the following: the category of the target in the bounding box, the confidence score of the bounding box, the width and height of the bounding box, and the corner position coordinates of the bounding box.
[0097] In the preprocessing of the detection model: preprocess the input image, such as operations like resizing and normalization, so as to input it into the detection model. The input of the model includes the input image captured by the fisheye camera, and the detection includes the model for target detection. The output of the model includes: a set of detected target bounding boxes, b i =(x i ,y i ,w i ,h i ), representing the i-th bounding box, where (x i ,y i ) are the coordinates of the upper left corner of the bounding box, w i and h i are the width and height of the bounding box respectively, c i is the category of the target in the i-th bounding box, and s i is the confidence score of the i-th bounding box.
[0098] In an embodiment of the present application, the fisheye camera is undistorted according to the distortion matrix of the fisheye camera to obtain an undistorted fisheye camera image, including: mapping the pixels in the original image in the fisheye camera to the corrected pixel positions according to the distortion matrix of the fisheye camera to obtain the undistorted fisheye camera image.
[0099] For the distortion matrix of the fisheye image that has been obtained, map the pixels in the original image in the fisheye camera to the corrected pixel positions, thereby obtaining the undistorted image, as shown in Figure 2 shown.
[0100] In an embodiment of the present application, the first center point parameter of the current target bounding box is compensated by the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box, including:
[0101] Calculate the normalized gradient direction:
[0102] flag x =-dx / (abs(dx) + 1e-6)
[0103] flag y =-dy / (abs(dy) + 1e-6)
[0104] where dx and dy are the offsets of the center point of the target box relative to the center point of the image when calculating the distance between the BBOX and the image center, so the obtained flag x and flag y represent the direction relative to the exact center of the image.
[0105] The formula for the final center point:
[0106] out_center_x = bbox x + delta x × flag x × sigma3 + distortion x × flag x × sigma 4 .
[0107] out_center_y = bbox y + delta y × flag y × sigma3 + distortion y × flag y × sigma4.
[0108] Among them, delta_x and delta_y are the linear compensation adjustment values of the x and y latitudes when calculating the distance-based linear compensation value.
[0109] The distortion_x and distortion_y are the distortion rates of the x and y latitudes.
[0110] The sigma3 and sigma4 are hyperparameters for the adjustment degree.
[0111] An embodiment of the present application also provides an image processing method for a fisheye camera. Among them, the parameter determination method of the fisheye camera, the method includes:
[0112] Undistort the fisheye camera according to the distortion matrix of the fisheye camera to obtain an undistorted fisheye camera image;
[0113] Detect the target bounding box in the undistorted fisheye camera image according to the pre-trained detection model;
[0114] Determine the first compensation value according to the distance between any target bounding box and the image center;
[0115] Determine the second compensation value according to the distortion matrix of the fisheye camera;
[0116] Compensate the first center point parameter of the current target bounding box through the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box.
[0117] In some embodiments, the determining the first compensation value according to the distance between any target bounding box and the image center includes:
[0118] Initialize the image center point and the center point in the target bounding box;
[0119] Calculate the offset distance of the center point of the target bounding box relative to the image center point according to the image center point and the center point in the target bounding box;
[0120] Based on the offset distance of the center point of the target bounding box relative to the image center point, the width and height of any target bounding box, calculate the offset amount of the center point of the target bounding box and use it as the first compensation value.
[0121] In some embodiments, the determining the second compensation value according to the distortion matrix of the fisheye camera includes:
[0122] Calculate the distance from the target original coordinate point of the original image of the fisheye camera to the center point coordinate of the original image;
[0123] Calculate the pixel coordinates after image undistortion according to the distortion matrix of the fisheye camera;
[0124] Obtain a distortion rate based on the pixel coordinates after undistorting the image and the target original coordinate point of the original image, and use it as the second compensation value.
[0125] In some embodiments, the distortion matrix of the fisheye camera includes:
[0126] Obtain the target pixel coordinates in the original image of the fisheye camera;
[0127] Determine the distance from the target pixel coordinates to the center point of the image;
[0128] Determine the radial distortion coefficient and tangential distortion coefficient according to the internal parameters of the fisheye camera;
[0129] Calculate the corrected pixel coordinates based on the target pixel coordinates in the original image of the fisheye camera, the distance from the target pixel coordinates to the center point of the image, the radial distortion coefficient, and the tangential distortion coefficient, and use them as the distortion matrix of the fisheye camera.
[0130] In some embodiments, detecting the target bounding box in the undistorted fisheye camera image according to the pre-trained detection model includes:
[0131] Detect a set of target bounding boxes of the undistorted fisheye camera image according to the pre-trained detection model, and the set of target bounding boxes includes at least one of the following: the category of the target in the bounding box, the confidence score of the bounding box, the width and height of the bounding box, and the corner position coordinates of the bounding box.
[0132] In some embodiments, undistorting the fisheye camera according to the distortion matrix of the fisheye camera to obtain an undistorted fisheye camera image includes:
[0133] Map the pixels in the original image in the fisheye camera to the corrected pixel positions according to the distortion matrix of the fisheye camera to obtain an undistorted fisheye camera image.
[0134] The embodiment of the present application also provides a parameter determination device 400 for a fisheye camera, as Figure 4 shown, which provides a schematic structural diagram of the parameter determination device for the fisheye camera in the embodiment of the present application. The parameter determination device 400 for the fisheye camera at least includes: an undistortion module 410, a bounding box detection module 420, a first compensation determination module 430, a second compensation determination module 440, and a compensation module 450, where:
[0135] In an embodiment of the present application, the undistortion module 410 is specifically configured to: undistort the fisheye camera according to the distortion matrix of the fisheye camera to obtain an undistorted fisheye camera image.
[0136] In the road-end autonomous driving solution, the fish-eye camera is mainly used to monitor the blind area under the pole. However, due to the image distortion of the fish-eye camera and different distortions in different areas, if the center point of the target image is used as the target center point in a simple way, a large deviation will occur in the position of the target center point.
[0137] The distortion matrix of the fish-eye camera can be obtained by relevant calculations, which is not specifically limited in the embodiments of the present application. What is obtained after undistorting the fish-eye camera according to the obtained distortion matrix of the fish-eye camera is the undistorted fish-eye camera image. As Figure 1 shown, it is a schematic diagram before the fish-eye camera is undistorted, and as Figure 2 shown, it is a schematic diagram after the fish-eye camera is undistorted.
[0138] In an embodiment of the present application, the boundary box detection module 420 is specifically configured to: detect the target boundary box in the undistorted fish-eye camera image according to a pre-trained detection model.
[0139] The pre-trained detection model can use a target detection model well-known in the relevant art or in the scenario, which is not specifically limited in the embodiments of the present application, as long as it can meet the function of target detection.
[0140] Detect the target boundary box BBOX in the undistorted fish-eye camera image through the pre-trained detection model. Exemplarily, the results output by the model include multiple target boundary boxes, a set of target boundary boxes obtained from the multiple target boundary boxes, and each target boundary box includes but is not limited to the width and height of the display boundary box, the target category in the boundary box such as a motor vehicle, a pedestrian or a non-motor vehicle, or a car, a large truck, a bus. In addition, the target boundary box also includes a confidence score. It can be understood that the confidence score is used as the basis or weight coefficient for target boundary box selection.
[0141] In an embodiment of the present application, the first compensation determination module 430 is specifically configured to: determine a first compensation value according to the distance between any target boundary box and the image center.
[0142] The "first compensation value" is used as a linear compensation value and can be used to make a more reasonable compensation for the subsequent center point calculation. It should be noted that the process of determining the "first compensation value" has nothing to do with whether the image is undistorted or whether the image has been detected by a pre-trained detection model. That is to say, the process of determining the "first compensation value" can be an independent process or can be determined in advance.
[0143] Exemplarily, first calculate the width and height of the target boundary box BBOX according to the distortion matrix of the fish-eye camera, then calculate the offset of the center point of the target box, and finally obtain the linear compensation adjustment values in the x and y dimensions.
[0144] In an embodiment of the present application, the second compensation determination module 440 is specifically configured to: determine a second compensation value according to the distortion matrix of the fisheye camera.
[0145] The "second compensation value", as the distortion rate, can be used to more reasonably compensate the calculation of the center point later. It should be noted that the process of determining the "second compensation value" has nothing to do with whether the image is undistorted or whether the image has been detected by a pre-trained detection model. That is to say, the process of determining the "second compensation value" can be an independent process or can be determined in advance.
[0146] Exemplarily, first calculate the distance from the original point (any position) in any target bounding box to the center point, and then calculate the pixel coordinates after undistortion. Then, based on the original point and the pixel coordinates after undistortion, finally, the distortion rate in the x dimension and the distortion rate in the y dimension can be calculated.
[0147] In an embodiment of the present application, the compensation module 450 is specifically configured to: compensate the first center point parameter of the current target bounding box through the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box.
[0148] Compensate the first center point parameter of the current target bounding box through the first compensation value and the second compensation value respectively, and finally obtain the second center point parameter of the current target bounding box. It can be understood that the "first center point parameter" refers to the center point position coordinates determined initially, and the "second center point parameter" refers to the center point position coordinates obtained after being compensated by the compensation value. At this time, the obtained center point position coordinates are more accurate and have smaller errors compared to the center point position coordinates obtained initially.
[0149] In an embodiment of the present application, the first compensation determination module 530 is further configured to
[0150] Initialize the center point of the image and the center point in the target bounding box;
[0151] Calculate the offset distance of the center point of the target bounding box relative to the center point of the image according to the center point of the image and the center point in the target bounding box;
[0152] Based on the offset distance of the center point of the target bounding box relative to the center point of the image, the width and height of any target bounding box, calculate the offset amount of the center point of the target bounding box and use it as the first compensation value.
[0153] In an embodiment of the present application, the second compensation determination module 540 is further configured to
[0154] Calculate the distance from the target original coordinate point of the original image of the fish-eye camera to the center point coordinate of the original image;
[0155] Calculate the pixel coordinates after image undistortion according to the distortion matrix of the fish-eye camera;
[0156] Obtain the distortion rate based on the pixel coordinates after the image is undistorted and the target original coordinate point of the original image, and use it as the second compensation value.
[0157] In an embodiment of the present application, the distortion matrix of the fish-eye camera includes:
[0158] Obtain the target pixel coordinates in the original image of the fish-eye camera;
[0159] Determine the distance from the target pixel coordinates to the center point of the image;
[0160] Determine the radial distortion coefficient and the tangential distortion coefficient according to the internal parameters of the fish-eye camera;
[0161] Calculate the corrected pixel coordinates based on the target pixel coordinates in the original image of the fish-eye camera, the distance from the target pixel coordinates to the center point of the image, the radial distortion coefficient, and the tangential distortion coefficient, and use them as the distortion matrix of the fish-eye camera.
[0162] In an embodiment of the present application, the bounding box detection module 520 is further configured to
[0163] Detect the set of target bounding boxes of the undistorted fish-eye camera image according to a pre-trained detection model, and the set of target bounding boxes includes at least one of the following: the category of the target in the bounding box, the confidence score of the bounding box, the width and height of the bounding box, and the corner position coordinates of the bounding box.
[0164] In an embodiment of the present application, the undistortion module 510 is further configured to
[0165] Map the pixels in the original image in the fish-eye camera to the corrected pixel positions according to the distortion matrix of the fish-eye camera to obtain the undistorted fish-eye camera image.
[0166] It can be understood that the above-mentioned parameter determination device of the fish-eye camera can implement each step of the fish-eye camera parameter determination method provided in the foregoing embodiments. The relevant explanations regarding the fish-eye camera parameter determination method are applicable to the fish-eye camera parameter determination device and will not be elaborated here.
[0167] Figure 5 It is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 5, at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.
[0168] The processor, network interface, and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 5 only a bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0169] The memory is used to store programs. Specifically, the program can include program code, and the program code includes computer operation instructions. The memory can include a memory and a non-volatile memory, and provide instructions and data to the processor.
[0170] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a parameter determination device for the fisheye camera at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:
[0171] Undistort the fisheye camera according to the distortion matrix of the fisheye camera to obtain an undistorted fisheye camera image;
[0172] Detect the target bounding box in the undistorted fisheye camera image according to the pre-trained detection model;
[0173] Determine a first compensation value according to the distance between any target bounding box and the image center;
[0174] Determine a second compensation value according to the distortion matrix of the fisheye camera;
[0175] Compensate the first center point parameter of the current target bounding box through the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box.
[0176] The above is as in this application Figure 3The method executed by the parameter determination device of the fisheye camera disclosed in the illustrated embodiment can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by the hardware decoding processor, or executed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.
[0177] The electronic device can also execute Figure 3 the method executed by the parameter determination device of the fisheye camera in Figure 3 the illustrated embodiment, and implement the functions of the parameter determination device of the fisheye camera in
[0178] The embodiments of the present application also propose a computer-readable storage medium. The computer-readable storage medium stores one or more programs. The one or more programs include instructions that, when executed by an electronic device including a plurality of application programs, can enable the electronic device to execute Figure 3 the method executed by the parameter determination device of the fisheye camera in the illustrated embodiment, and specifically used to execute:
[0179] Undistort the fisheye camera according to the distortion matrix of the fisheye camera to obtain the undistorted fisheye camera image;
[0180] Detect the target bounding box in the undistorted fisheye camera image according to the pre-trained detection model;
[0181] Determine a first compensation value according to the distance between an arbitrary target bounding box and the center of the image;
[0182] Determine a second compensation value according to the distortion matrix of the fisheye camera;
[0183] Compensate the first center point parameter of the current target bounding box by the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box.
[0184] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0185] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.
[0186] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that realizes the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.
[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for realizing the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.
[0188] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0189] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0190] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0191] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0192] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0193] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.
Claims
1. A method for determining parameters of a fisheye camera, wherein: The method comprises: Dedistorting the fisheye camera according to the distortion matrix of the fisheye camera to obtain a dedistorted fisheye camera image; Detecting a target bounding box in the dedistorted fisheye camera image according to a pre-trained detection model; Determining a first compensation value according to a distance between an arbitrary target bounding box and a center of the image; Determining a second compensation value according to a distortion matrix of the fisheye camera; The first center point parameter of the current target bounding box is compensated by the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box.
2. The method of claim 1, wherein: The determining of the first compensation value according to the distance between the arbitrary target bounding box and the image center includes: Initialize the center point of the image and the center point of the target bounding box; Calculate the offset distance of the center point of the target bounding box relative to the center point of the image according to the center point of the image and the center point in the target bounding box; Based on the offset distance of the center point of the target bounding box relative to the center point of the image, and the width and height of any target bounding box, the offset of the center point of the target bounding box is calculated and used as the first compensation value.
3. The method of claim 1, wherein: The determining of the second compensation value according to the distortion matrix of the fisheye camera includes: Calculate the distance between the target original coordinate point of the original image of the fisheye camera and the coordinate of the center point of the original image; Calculating pixel coordinates of the dedistorted image according to the distortion matrix of the fisheye camera; The distortion rate is obtained according to the pixel coordinates of the dedistorted image and the target original coordinate points of the original image and is used as the second compensation value.
4. The method of claim 3, wherein: The distortion matrix of the fisheye camera includes: Get the target pixel coordinates in the original image of the fisheye camera; Determine the distance from the target pixel coordinate to the center point of the image; Determine the radial distortion coefficient and the tangential distortion coefficient according to the internal parameters of the fisheye camera; According to the target pixel coordinates in the original image of the fisheye camera, the distance from the target pixel coordinates to the center point of the image, the radial distortion coefficient and the tangential distortion coefficient, the corrected pixel coordinates are calculated and used as the distortion matrix of the fisheye camera.
5. The method of claim 1, wherein: The detecting the target bounding box in the dedistorted fisheye camera image according to the pre-trained detection model includes: A target bounding box set of the dedistorted fisheye camera image is obtained by detection according to a pre-trained detection model, wherein the target bounding box set includes at least one of the following: a category of the target in the bounding box, a confidence score of the bounding box, a width and height of the bounding box, and a corner point position coordinate of the bounding box.
6. The method of claim 1, wherein: Dedistorting the fisheye camera according to the distortion matrix of the fisheye camera to obtain a dedistorted fisheye camera image includes: According to the distortion matrix of the fisheye camera, the pixels in the original image of the fisheye camera are mapped to the corrected pixel positions to obtain the dedistorted fisheye camera image.
7. A method for processing an image of a fisheye camera, wherein: The method comprises the method for determining parameters of a fisheye camera as claimed in any one of claims 1 to 6.
8. A device for determining parameters of a fisheye camera, wherein: The device comprises: A dedistortion module, used for dedistorting the fisheye camera according to the distortion matrix of the fisheye camera to obtain a dedistorted fisheye camera image; A bounding box detection module, configured to detect a target bounding box in the dedistorted fisheye camera image according to a pre-trained detection model; A first compensation determination module, used to determine a first compensation value according to a distance between an arbitrary target bounding box and an image center; A second compensation determination module, used to determine a second compensation value according to the distortion matrix of the fisheye camera; The compensation module is used to compensate the first center point parameter of the current target bounding box by using the first compensation value and the second compensation value to obtain the second center point parameter of the current target bounding box.
9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.
10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.