A fisheye camera-based panoramic perception method, device, equipment and medium

Abstract

The application discloses a kind of panoramic perception method, device, equipment and medium based on fish-eye camera, can be widely applied in real-time look-around depth estimation field, method includes: four fish-eye cameras are obtained four fish-eye images;Four fish-eye cameras are placed on the four vertices of a square respectively, lens is directed to the outside of the diagonal of square;Each fish-eye camera generates two virtual pinhole cameras, each virtual pinhole camera and adjacent fish-eye camera generated virtual pinhole camera constitute binocular camera;According to the mapping table generated by the imaging principle of fish-eye camera and the parameter of virtual pinhole camera in advance, each fish-eye image is converted into the left image and right image of corresponding binocular camera;The left image feature of each binocular camera left image and the right image feature of right image are extracted using binary neural network;Disparity calculation is carried out to the left image feature and right image feature corresponding to each binocular camera, and four depth maps covering the panoramic of the position of four fish-eye cameras are obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of real-time surround view depth estimation, and in particular to a panoramic perception method based on a fisheye camera, an apparatus, a device and a medium. BACKGROUND

[0002] Fisheye cameras have serious non-linear distortion, and the image of the same object at different angles is inconsistent. There is no epipolar constraint in traditional stereo matching, which brings great challenges to stereo matching. Therefore, the fisheye surround stereo matching algorithm is usually complex and requires high computing power. Using multi-view fisheye cameras for surround stereo matching requires the calculation of multiple images. Since the algorithm for processing a single image has high computational complexity, the requirement for computing power is higher when processing multiple images, making it difficult to run in real time. Most current methods use deep neural networks to process multi-view fisheye stereo matching problems. When the actual use environment and the data set used to train the neural network have large differences, the network performance decreases, and the generalization is poor.

[0003] Therefore, the above problems need to be solved. SUMMARY

[0004] Therefore, the present application provides a panoramic perception method based on a fisheye camera, an apparatus, a device and a medium to improve the real-time running efficiency and generalization of fisheye camera surround stereo matching.

[0005] An aspect of the present application provides a panoramic perception method based on a fisheye camera, comprising:

[0006] Four fisheye cameras are used to obtain four corresponding fisheye images. The four fisheye cameras are placed at the four vertices of a square, and the lenses are directed outward from the diagonal of the square. Each fisheye camera generates two virtual pinhole cameras, and each virtual pinhole camera and the virtual pinhole camera generated by the adjacent fisheye camera form a binocular camera.

[0007] According to a mapping table generated in advance from the imaging principle of the fisheye camera and the parameters of the virtual pinhole camera, each fisheye image is converted into a left image and a right image corresponding to the binocular camera.

[0008] A binary neural network is used to extract left image features corresponding to the left image and right image features corresponding to the right image of each binocular camera.

[0009] Disparity calculation is performed on the left image features and right image features corresponding to each binocular camera to obtain four depth maps covering the positions of the four fisheye cameras.

[0010] Optionally, the generation process of the mapping table comprises:

[0011] determining parameters of a preset virtual pinhole camera;

[0012] projecting an image point in a virtual pinhole image captured by the virtual pinhole camera to a three-dimensional space according to the parameters of the virtual pinhole camera;

[0013] projecting the image point in the three-dimensional space to a fisheye camera imaging plane according to an imaging principle of the fisheye camera;

[0014] generating a mapping table of the image point and the fisheye camera imaging plane.

[0015] Optionally, the projecting the image point in the three-dimensional space to the fisheye camera imaging plane according to the imaging principle of the fisheye camera comprises:

[0016] converting the image point into a unit vector on a unit sphere;

[0017] calculating a radial distance relationship between a light ray incident angle and a center point of the fisheye camera imaging plane to a projection point of the fisheye camera imaging plane according to the imaging principle of the fisheye camera;

[0018] calculating a projection coordinate of the image point on the fisheye camera imaging plane according to the unit vector and the radial distance relationship.

[0019] Optionally, the method further comprises:

[0020] calibrating intrinsic parameters of the fisheye camera by using a least square optimization and according to polynomial parameters of the projection coordinate, an affine transformation matrix of the fisheye camera, a position coordinate of the center point of the fisheye camera imaging plane when the light ray incident angle is 0, and a rotation matrix and a translation matrix between the fisheye camera and a calibration board.

[0021] Optionally, the disparity calculation on the left image feature and the right image feature corresponding to each of the binocular cameras to obtain four depth maps covering a panorama of positions of the four fisheye cameras comprises:

[0022] the matching cost calculation on the left image feature and the right image feature corresponding to each of the binocular cameras to determine a right image pixel corresponding to a left image pixel of the left image feature in the right image feature;

[0023] the smoothing processing on all right image pixels corresponding to each of the left image pixels to aggregate a global matching cost;

[0024] the pixel displacement calculation on the right image pixels corresponding to each of the left image pixels in the global matching cost by using a semi-global matching algorithm to obtain a disparity value and determine a corresponding disparity map according to the disparity value;

[0025] determine four depth maps covering a panorama of positions of the four fisheye cameras according to the disparity map and parameters of the virtual pinhole cameras.

[0026] Optionally, the matching cost calculation of the left image features and the right image features corresponding to each of the binocular cameras comprises:

[0027] The Hamming distance is calculated as the matching cost calculation of the left image features and the right image features corresponding to each of the binocular cameras.

[0028] Optionally, before the determining of the four depth maps covering a panorama of positions of the four fisheye cameras according to the disparity map and parameters of the virtual pinhole cameras, the method further comprises:

[0029] At least one of disparity smoothing, disparity interpolation, disparity consistency checking or edge preservation is performed on the disparity map to obtain a disparity-optimized disparity map.

[0030] The determining of the four depth maps covering a panorama of positions of the four fisheye cameras according to the disparity map and parameters of the virtual pinhole cameras comprises:

[0031] The four depth maps covering a panorama of positions of the four fisheye cameras are determined according to the disparity-optimized disparity map.

[0032] Another aspect of the embodiment of the present application further provides a fisheye camera-based panoramic perception device, comprising:

[0033] a fisheye image acquisition unit configured to acquire four fisheye images corresponding to four fisheye cameras respectively; wherein the four fisheye cameras are placed at four vertices of a same square respectively, and the lenses are directed outward from the diagonal lines of the square; each fisheye camera generates two virtual pinhole cameras, and each virtual pinhole camera constitutes a binocular camera with a virtual pinhole camera generated by an adjacent fisheye camera;

[0034] a fisheye image conversion unit configured to convert each of the fisheye images into left and right images corresponding to the binocular cameras according to a mapping table generated in advance by imaging principles of the fisheye cameras and parameters of the virtual pinhole cameras;

[0035] an image feature extraction unit configured to extract left image features corresponding to the left images and right image features corresponding to the right images of each of the binocular cameras by using a binary neural network;

[0036] a disparity calculation unit configured to perform disparity calculation on the left image features and the right image features corresponding to each of the binocular cameras to obtain four depth maps covering a panorama of positions of the four fisheye cameras.

[0037] Another aspect of the embodiment of the present application also provides an electronic device, comprising a processor and a memory;

[0038] The memory is used for storing a program;

[0039] The processor executes the program to realize the fisheye camera-based panoramic perception method.

[0040] Another aspect of the embodiment of the present application also provides a computer readable storage medium, the storage medium stores a program, and the program is executed by a processor to realize the fisheye camera-based panoramic perception method.

[0041] The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the foregoing method.

[0042] The present application restores the epipolar constraint by converting the fisheye image into a pinhole-like image, i.e. the left image and the right image of the binocular camera composed of two virtual pinhole cameras, reduces the complexity of the algorithm, and reduces the requirement for computing power; the present application uses a binary neural network to extract image features, and quantization of the network significantly reduces the amount of calculation, and there is no obvious attenuation in accuracy; in addition, in the present application, the binary neural network can only be responsible for feature extraction, and has a single function, and therefore has good generalization, and is suitable for various use scenarios. Therefore, the present application can run in real time on a platform with limited computing power, such as an Xavier, an FPGA and the like, under the condition of maintaining accuracy, and has a wider application scenario. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0044] Figure 1 A flowchart of a fisheye camera-based panoramic perception method provided by the embodiment of the present application is shown in the figure;

[0045] Figure 2 A top view of a fisheye camera provided by the embodiment of the present application is shown in the figure;

[0046] Figure 3 A schematic diagram of a virtual pinhole binocular camera provided by the embodiment of the present application is shown in the figure;

[0047] Figure 4 An imaging process schematic diagram of a fisheye camera imaging model provided by an embodiment of the present application is shown in FIG. 1.

[0048] Figure 5 A process schematic diagram of refraction of light by a fisheye lens provided by an embodiment of the present application is shown in FIG. 2.

[0049] Figure 6 A schematic diagram of conversion of a fisheye image into a virtual pinhole image provided by an embodiment of the present application is shown in FIG. 3.

[0050] Figure 7 An example flowchart of stereo matching provided by an embodiment of the present application is shown in FIG. 4.

[0051] Figure 8 A comparison result diagram of an embodiment of the present application with an existing algorithm is shown in FIG. 5.

[0052] Figure 9 A converted virtual pinhole image and a generated disparity map provided by an embodiment of the present application are shown in FIG. 6.

[0053] Figure 10 A flowchart of a stereo matching algorithm provided by an embodiment of the present application is shown in FIG. 7.

[0054] Figure 11 An example flowchart of panoramic perception based on a fisheye camera provided by an embodiment of the present application is shown in FIG. 8.

[0055] Figure 12 A structural block diagram of a panoramic perception device based on a fisheye camera provided by an embodiment of the present application is shown in FIG. 9. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely used to explain the present application and do not limit the present application.

[0057] Referring to Figure 1 An embodiment of the present application provides a panoramic perception method based on a fisheye camera, which specifically includes the following steps:

[0058] S100: Four fisheye images are obtained by four fisheye cameras; wherein the four fisheye cameras are respectively placed at four vertices of a square, and the lenses are directed outward from the diagonal of the square; each fisheye camera generates two virtual pinhole cameras, and each virtual pinhole camera and a virtual pinhole camera generated by an adjacent fisheye camera form a binocular camera.

[0059] First, the fisheye camera of the embodiment of the present application is described with reference to Figure 2The embodiment of the present application provides a top view of a fisheye camera.

[0060] The fisheye camera of the embodiment of the present application can have a maximum shooting angle of 220°, and other fisheye cameras with other angles can also be selected according to actual conditions. The four fisheye cameras are horizontally placed at four vertices of a square, and the lenses are directed outward along the diagonal of the square, so that the images captured by the four fisheye cameras can cover the surrounding scene by 360°, that is, the panoramic image of the surrounding can be obtained.

[0061] Specifically, each fisheye camera can generate two virtual pinhole cameras, each virtual pinhole camera and the virtual pinhole camera generated by the adjacent fisheye camera can constitute a binocular camera, and the four fisheye cameras can generate eight virtual pinhole cameras to constitute four groups of binocular cameras to cover 360° horizontally. Figure 3 As shown in the figure, the embodiment of the present application provides a schematic diagram of a virtual pinhole binocular camera. The diagonal direction of the square can represent the fisheye camera and the optical center direction of the fisheye camera, and the two virtual pinhole cameras on the same side of the square constitute a group of binocular cameras.

[0062] S110: According to the mapping table generated in advance by the imaging principle of the fisheye camera and the parameters of the virtual pinhole camera, each fisheye image is converted into a left image and a right image corresponding to the binocular camera.

[0063] Specifically, the generation process of the mapping table can include:

[0064] S1, determining the parameters of the virtual pinhole camera preset in advance.

[0065] Specifically, since the virtual pinhole camera is virtually generated, its parameters can be freely set by the user.

[0066] S2, projecting the image points in the virtual pinhole image obtained by the virtual pinhole camera into a three-dimensional space according to the parameters of the virtual pinhole camera.

[0067] S3, projecting the image points in the three-dimensional space into the imaging plane of the fisheye camera according to the imaging principle of the fisheye camera.

[0068] S4, generating the mapping table of the image points and the imaging plane of the fisheye camera.

[0069] Specifically, the above steps S2 and S3 are described. First, the fisheye camera imaging model is described as follows:

[0070] The fisheye camera imaging model can be used to project the three-dimensional coordinates in space, that is, the coordinates of the image points, into two-dimensional pixel coordinates of the fisheye image. The imaging process of the fisheye camera imaging model can refer to Figure 4 .

[0071] Specifically, Figure 4 P C represent a point in three-dimensional space, S represents a unit sphere, Ω represents an imaging plane of a fisheye camera, O C , O Cl are two virtual points, which abstractly represent an incident point and an exit point of a light ray entering the fisheye lens and being refracted, z passes through the center principal point O i of the imaging plane of the fisheye camera vertically upward, representing the optical axis of the lens.

[0072] The process of refraction of light by the fisheye lens can be referred to Figure 5 , and the process can have the following properties: (1) any light ray passing through the optical axis will not change direction; (2) as the incident angle θ i increases, the exit angle θ r monotonically increases; (3) if the incident light rays are radially symmetric, then the refracted light rays are also radially symmetric, which means that if the incident angle θ i is the same, then the exit angle θ r is also the same.

[0073] Use R(θ) to represent the radial distance relationship between the incident angle θ and the projection point u of the imaging plane principal point O i to the imaging plane, that is, R(θ) = |O i u|, and abstractly simplify the light refraction process to obtain R ref (θ), which represents the reference radial distance related to the incident angle θ. Use the equidistance model as the reference radial distance, that is, R ref (θ) = fθ, where f is the focal length.

[0074] The lens distortion formula is defined as follows:

[0075]

[0076] The distortion function D(θ) can be implemented by a high-order polynomial, and according to the above formula, the distance from the plane projection point to the imaging plane principal point is:

[0077]

[0078] Therefore, the step S3 of projecting the image point in the three-dimensional space to the imaging plane of the fisheye camera according to the imaging principle of the fisheye camera can include:

[0079] S31, converting the image point into a unit vector on the unit sphere.

[0080] S32, calculating the radial distance relationship between the incident angle of the light ray and the projection point of the center point of the imaging plane of the fisheye camera to the imaging plane of the fisheye camera according to the imaging principle of the fisheye camera.

[0081] S33, calculating the projection coordinates of the image point on the fisheye camera imaging plane according to the unit vector and the radial distance relationship.

[0082] Specifically, as shown in Figure 4 , the projection of a three-dimensional world coordinate point onto the fisheye camera imaging plane can include the following steps: transforming the three-dimensional point into a unit vector; according to Figure 5 , calculating R(θ) by the process of refraction of the fisheye lens to light; decomposing R(θ) to obtain the projection coordinates of the image point u.

[0083] (1) Transform the three-dimensional point into a unit vector.

[0084] Assume P c = [x C , y C , z C ] T , first convert it to the According to the definition of the coordinate system in the figure, two angle values can be obtained:

[0085]

[0086]

[0087] (2) According to the process of refraction of the fisheye lens to light shown in Figure 5 , calculate R(θ).

[0088] According to the formula of refraction of the fisheye camera to light, we can get:

[0089]

[0090] Where R(θ) = fR0(θ).

[0091] (3) Decompose R(θ) to obtain the projection coordinates of the image point u.

[0092] The imaging plane coordinates are:

[0093]

[0094] Where [c x , c y ] T represents the principal point position when the incident angle is 0. The focal length f is replaced by the 2x2 affine transformation matrix A f to handle changes such as rotation between the lens and the sensor. By integrating the above formulas, the projection formula can be obtained:

[0095]

[0096] wherein

[0097] So far, the fisheye imaging model is explained.

[0098] Then, the process of converting each fisheye image into the left image and the right image of the corresponding binocular camera according to the mapping table is described.

[0099] Specifically, since a virtual pinhole camera is generated, the camera parameters can be customized. After setting the pinhole camera parameters, the image points are projected into the three-dimensional space, and then the three-dimensional points are projected onto the fisheye camera imaging plane by using the above projection formula. The process is expressed by the formula as follows:

[0100]

[0101] wherein u V represents the virtual pinhole image pixel coordinates, represents the projection of the virtual pinhole image from two dimensions to three dimensions, represents the rotation matrix between the virtual pinhole camera and the fisheye camera, π C represents the projection of the fisheye camera, u C represents the fisheye image pixel coordinates. The above completes the process from the 2D pinhole image plane -> spatial three-dimensional point -> fisheye imaging plane. In an optional implementation, in actual use, the fisheye image can be obtained, and the goal is to generate a virtual pinhole image, i.e., the left image and the right image of the binocular camera. Therefore, the mapping table is generated by the above process, and the conversion from the fisheye image to the virtual pinhole image is completed by using the reamp() function in OpenCV. Referring to Figure 6 , the embodiment of the present application provides a schematic diagram of converting a fisheye image into a virtual pinhole image.

[0102] Further, the embodiment of the present application can further include:

[0103] The intrinsic parameters of the fisheye camera are calibrated by using the least square optimization and according to the polynomial parameters of the projection coordinates, the affine transformation matrix of the fisheye camera, the position coordinates of the center point of the fisheye camera imaging plane when the light incidence angle is 0, and the rotation matrix and the translation matrix between the fisheye camera and the calibration board.

[0104] Specifically, the parameters to be calibrated include η i the polynomial parameters, A f the matrix, and c x , c y . The re-projection error is defined as follows:

[0105]

[0106] wherein and For the rotation matrix and translation matrix between the camera and the calibration board, the PNP algorithm can be used to obtain u, which is the image point coordinate, and the target is to minimize this error, and the intrinsic parameters of the fisheye camera can be obtained by using the least square optimization.

[0107] S120: Extract left image features corresponding to each left image and right image features corresponding to each right image of the binocular camera using a binary neural network.

[0108] Specifically, with the application of deep neural networks, the stereo matching research has made great progress in accuracy. However, the problem caused by high accuracy is large computational overhead. The current most advanced stereo matching algorithm needs a long time to calculate the depth from a single frame of binocular image, even on the most powerful GPU. At the same time, since the traditional deep learning uses full-precision floating-point calculation to extract features from the picture, and uses the floating-point cosine similarity calculation of the feature matrix, there is a serious conflict between these complex operations and limited computing resources, which is difficult to run in real time on platforms with limited computing power.

[0109] To solve this problem, the embodiment of the application can use binary quantization to quantize the traditional neural network. This design not only reduces the storage resources, but also reduces the computational complexity, so that the platform with limited computing power can be deployed and run, and accurate scene depth information can be obtained with small storage space and short running time.

[0110] Binary neural network (BNN) refers to quantizing the parameters and inputs of a deep neural network to 1 bit, and the data only has two possible values of 1 and 0 (or -1). After using binary quantization to compress the network, the weights and activation values in the network only need a small amount of storage space, and at the same time, the binary network can use lightweight bit operations instead of floating-point operations, which has the characteristics of hardware friendliness.

[0111] S130: Disparity calculation is performed on the left image features and right image features corresponding to each binocular camera to obtain four depth maps covering the positions of the four fisheye cameras.

[0112] Specifically, the disparity calculation process is a disparity estimation process, and the disparity estimation can also be called stereo matching. The input of the disparity estimation is a pair of left and right images captured at the same time and subjected to epipolar rectification. The output is an image composed of disparity values corresponding to each pixel in the reference image. Disparity is the pixel-level difference between the corresponding points in the left and right images of a point in a three-dimensional scene. After obtaining the disparity, the distance from the camera to the surrounding objects can be calculated using the camera intrinsic and extrinsic parameters.

[0113] Referring to Figure 7The embodiment of the application provides a flowchart example of stereo matching.

[0114] Next, the process will be described. Figure 7 The process will be described.

[0115] 1. Matching cost calculation.

[0116] Matching cost calculation refers to measuring the similarity or difference between left and right pixels to determine the corresponding right pixel for each left pixel. The purpose of matching cost calculation is to find the best right pixel corresponding to each pixel in the left image, thereby determining the disparity value of the pixel. Common matching cost calculation methods include the following: SSD (Sum of Squared Differences), the sum of the squares of the differences between left and right image pixels; NCC (Normalized Cross Correlation), the normalized cross-correlation coefficient between left and right pixels; SAD (Sum of Absolute Differences), the sum of the absolute values of the differences between left and right pixels.

[0117] 2. Cost aggregation.

[0118] Cost aggregation refers to smoothing the matching cost of all right pixels corresponding to the left pixel to reduce false matches and discontinuous disparity results. The purpose of cost aggregation is to combine local cost information to form a global matching cost and provide more accurate information for subsequent disparity calculation. Cost aggregation can effectively reduce false matches and discontinuous disparity results and provide more accurate information for subsequent disparity calculation.

[0119] 3. Disparity calculation.

[0120] Disparity calculation refers to determining the pixel displacement, i.e. disparity, of the right pixel corresponding to each left pixel based on the matching cost between left and right pixels. Disparity is a measure of the distance difference between different pixels in the camera's field of view and is also an indicator of object depth. According to the matching cost between left and right pixels, the best matching right pixel corresponding to each left pixel can be obtained, thereby calculating the disparity value of each pixel. Disparity calculation is a core step in stereo matching, and its accuracy directly affects the effect of stereo matching.

[0121] 5. Disparity optimization.

[0122] Disparity optimization refers to fine-tuning the initial disparity map to improve the quality and accuracy of the disparity map. The purpose of disparity optimization is to improve the accuracy and stability of the disparity map for subsequent depth calculation and three-dimensional reconstruction applications. Disparity optimization usually includes the following aspects: disparity smoothing, disparity interpolation, disparity consistency checking, edge preservation, etc.

[0123] 6. Hamming distance.

[0124] Hamming distance is a distance measurement used in error detection coding for digital transmission. It represents the number of positions at which corresponding bits are different. XOR operation is performed on two strings, and the number of 1s in the result is counted, which is the Hamming distance. We can also understand the Hamming distance as the minimum number of substitutions needed to change one of the two equal-length strings into the other.

[0125] 7. Semi-global matching algorithm.

[0126] Semi-global matching algorithm (SGM) is a commonly used algorithm for calculating disparity map, which is often used in the field of stereo matching and three-dimensional reconstruction, and improves the performance of global matching algorithm. The core idea of SGM is to transform the stereo matching problem into an energy minimization problem, and calculate the cost cube of each pixel by calculating the cost convolution, and then calculate the disparity value of the pixel by cost aggregation. In the process of cost aggregation, SGM introduces global smoothness constraint, which makes the calculated disparity map more smooth and reduces the false matching situation. Compared with traditional global matching algorithm, SGM algorithm has higher precision and lower computational complexity, so it has been widely used in the field of stereo matching and three-dimensional reconstruction.

[0127] Therefore, the above step S130, for each left image feature and right image feature corresponding to the binocular camera, disparity calculation is performed to obtain four depth maps covering the positions of the four fisheye cameras. It can include:

[0128] S1, the matching cost of the left image feature and the right image feature corresponding to each binocular camera is calculated to determine the right image pixel corresponding to the left image pixel of the left image feature in the right image feature.

[0129] Specifically, the Hamming distance of the left image feature and the right image feature corresponding to each binocular camera can be calculated as the matching cost calculation.

[0130] S2, all right image pixels corresponding to each left image pixel are smoothed to aggregate the global matching cost.

[0131] S3, the pixel displacement of the right image pixel corresponding to each left image pixel in the global matching cost is calculated by using the semi-global matching algorithm to obtain the disparity value, and the corresponding disparity map is determined according to the disparity value.

[0132] S4, determining four depth maps covering the panorama of the positions of the four fisheye cameras according to the disparity map and the parameters of the virtual pinhole camera.

[0133] Further, in order to optimize the disparity map, the embodiment of the present application can further comprise: performing at least one of disparity smoothing, disparity interpolation, disparity consistency checking or edge preservation on the disparity map to obtain a disparity-optimized disparity map. Then, four depth maps covering the panorama of the positions of the four fisheye cameras are determined according to the disparity-optimized disparity map.

[0134] Specifically, according to experimental results and theoretical analysis, it is found that the way of obtaining a depth map only by cost calculation is difficult to adapt to complex actual environments. Therefore, some semi-global and global algorithms are applied to the correction of the above-mentioned cost to improve the final output accuracy. This step is called cost aggregation. In order to improve the robustness of the algorithm, the embodiment of the present application can use a semi-global matching (SGM) algorithm to perform cost aggregation. The SGM algorithm sets an energy function for the disparity map, and the optimal disparity value of each pixel is obtained by minimizing the energy function. Finally, the disparity value of each pixel is determined according to the cost aggregation matrix and the disparity map is output. Moreover, the SGM algorithm used in the embodiment of the present application is optimized by parallel acceleration, which can improve the accuracy of the operation result with very little running time.

[0135] According to the experimental results, the binary network stereo matching algorithm used in the embodiment of the present application can achieve a competitive accuracy rate under the premise of obtaining a great speed-up effect, Figure 8 The results of the comparison of the frame rate and error of the algorithm with the existing algorithm are shown in the following figure.

[0136] Referring to Figure 9 The embodiment of the present application provides a converted virtual pinhole image and a generated disparity map, wherein, Figure 9 (a) and (b) in the figure are two virtual pinhole images, and (c) is a generated disparity map.

[0137] Referring to Figure 10 The embodiment of the present application provides a stereo matching algorithm flowchart. The input of stereo matching is a converted virtual pinhole image. The binary neural network is used to extract image features. After the matching cost is calculated, the semi-global matching algorithm (SGM) is used for cost aggregation, and the disparity image is calculated.

[0138] The stereo matching algorithm of the embodiment of the present application adopts a non-end-to-end method, and the binary network is trained by simultaneously binarizing the weight and the activation value. Meanwhile, the Hamming distance with hardware friendliness is used to compare and calculate the matching cost of the features.

[0139] Next, the application process of the present application will be described with specific examples.

[0140] Referring to Figure 11 An example flowchart of panoramic perception based on fisheye cameras is provided in embodiments of the present application.

[0141] Specifically, embodiments of the present application relate to a real-time surround view depth estimation technique based on fisheye cameras. The technique uses four fisheye cameras to capture images, and through a rectification algorithm, each fisheye image is converted into two virtual pinhole images. Through reasonable configuration design, each virtual pinhole image is stereoscopically opposite to the virtual pinhole image obtained by rectification of the adjacent fisheye camera. The stereoscopic opposite is input into a binary feature extraction network to obtain a binary feature map of each image. Then, the Hamming distance is used to calculate the cost between the two images, and the cost is aggregated through a semi-global matching algorithm to obtain a depth map. The four fisheye cameras can generate four sets of stereoscopic corresponding relationships covering a horizontal 360° direction, so the technique can realize real-time surround view depth estimation.

[0142] Embodiments of the present application use a rectification algorithm to rectify one fisheye image into two virtual pinhole images in the horizontal direction, which reduces the difficulty of fisheye stereo matching, greatly simplifies the complexity of the fisheye stereo matching algorithm, and reduces the demand for computing power. The virtual pinhole images are stereo-matched using a quantized binary neural network, which greatly reduces the time-consuming of calculation while ensuring accuracy, and can run in real time on devices with limited computing power. In addition, since the network function used is single and only responsible for feature extraction, and has been scientifically and reasonably trained, it has good generalization and is suitable for more scenarios.

[0143] In addition, embodiments of the present application use a four-eye fisheye camera to realize panoramic perception. Compared with the scheme of realizing panoramic perception using multiple pinhole cameras, embodiments of the present application greatly reduce the volume of the panoramic perception system module, making it more convenient to deploy. Compared with single-view environmental perception, even if the camera in one direction of the panoramic perception system is blocked or disabled due to light, the panoramic perception system can still maintain its environmental perception ability based on the information provided by the cameras in other directions.

[0144] Referring to Figure 12 An example panoramic perception device based on fisheye cameras is provided in embodiments of the present application.

[0145] The fisheye image acquisition unit is configured to acquire four fisheye images through four fisheye cameras; wherein the four fisheye cameras are respectively placed at the four vertices of a square, and the lenses are directed outward from the diagonal of the square; each fisheye camera generates two virtual pinhole cameras, and each virtual pinhole camera and the virtual pinhole camera generated by the adjacent fisheye camera form a binocular camera;

[0146] a fisheye image conversion unit, configured to convert each of the fisheye images into left images and right images corresponding to the binocular cameras according to a mapping table generated in advance by the imaging principle of the fisheye cameras and the parameters of the virtual pinhole cameras;

[0147] an image feature extraction unit, configured to extract left image features corresponding to the left images and right image features corresponding to the right images of each of the binocular cameras by using a binary neural network;

[0148] a disparity calculation unit, configured to perform disparity calculation on the left image features and the right image features corresponding to each of the binocular cameras to obtain four depth maps covering a panorama of positions of the four fisheye cameras.

[0149] The embodiment of the present application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method shown in the embodiment. Figure 1 The method shown in the embodiment.

[0150] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, and the purpose is to provide a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0151] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary to an understanding of the application. Rather, the actual implementation of the modules, in combination with their attributes, functions, and internal relationships, are to be understood within the context of the devices disclosed herein. Thus, those skilled in the art with access to patents, scientific journals, and other public sources known by those skilled in the art will be able to practice the application as set forth in the claims without undue experimentation, using ordinary skill in the art along with the present disclosure. It is also to be understood that the specific concepts disclosed are merely illustrative and that the scope of the present application is to be determined by the entire scope of the claims, along with all equivalents of the claims and their equivalents.

[0152] If the functions are implemented in software, the functions can be stored in or implemented as one or more computer program products, which can be incorporated into a computer-readable medium for use by or in connection with an apparatus, method or system as described herein. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, an article of manufacture or a computer program product. The computer-readable medium can be a non-transitory computer-readable medium. The computer-readable medium can be a computer-readable non-transitory storage medium. The computer-readable medium can be a tangible computer-readable medium. The computer-readable medium can be a computer-readable tangible non-transitory medium.

[0153] Logic and / or steps represented in flow diagrams and / or described herein, for example, can be embodied in non-transitory computer-readable media, which can be executed by an instruction execution system, apparatus, or device, such as a computer-based system, processor, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium. The computer-readable storage medium can be a tangible computer- readable storage medium.

[0154] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0155] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques can be used to implement the hardware used in the described embodiments: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0156] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In the specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific feature, structure, material or characteristic described can be combined in any appropriate manner in one or more embodiments or examples.

[0157] Although embodiments of the application have been shown and described, it would be recognized by those of ordinary skill in the art that various changes, modifications, substitutions, and alterations can be made hereto without departing from the spirit and scope of the application. The scope of the application is limited only by the claims and the equivalents thereof.

[0158] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A fisheye camera based panoramic perception method, characterized in that, The method comprises: four fisheye cameras are arranged at four vertices of a square, and the lenses of the fisheye cameras are directed outward from the diagonal of the square; each fisheye camera generates two virtual pinhole cameras, and each virtual pinhole camera and a virtual pinhole camera generated by an adjacent fisheye camera form a binocular camera; each fisheye image is converted into a left image and a right image corresponding to the binocular camera according to a mapping table generated in advance according to the imaging principle of the fisheye camera and the parameters of the virtual pinhole camera; left image features corresponding to the left image and right image features corresponding to the right image of each binocular camera are extracted by using a binary neural network; disparity calculation is performed on the left image features and the right image features corresponding to each binocular camera to obtain four depth maps covering the panorama of the positions of the four fisheye cameras; the disparity calculation on the left image features and the right image features corresponding to each binocular camera to obtain four depth maps covering the panorama of the positions of the four fisheye cameras comprises: matching cost calculation is performed on the left image features and the right image features corresponding to each binocular camera to determine right pixels corresponding to left pixels of the left image features in the right image features; all right pixels corresponding to each left pixel are smoothed to obtain a global matching cost; pixel displacement of the right pixels corresponding to each left pixel in the global matching cost is calculated by using a semi-global matching algorithm to obtain a disparity value, and a corresponding disparity map is determined according to the disparity value; four depth maps covering the panorama of the positions of the four fisheye cameras are determined according to the disparity map and the parameters of the virtual pinhole camera.

2. The fisheye camera based panoramic perception method of claim 1, wherein, The generation process of the mapping table comprises: parameters of the virtual pinhole camera are determined in advance; image points in a virtual pinhole image captured by the virtual pinhole camera are projected to a three-dimensional space according to the parameters of the virtual pinhole camera; the image points in the three-dimensional space are projected to a fisheye camera imaging plane according to the imaging principle of the fisheye camera; the mapping table of the image points and the fisheye camera imaging plane is generated.

3. The fisheye camera based panoramic perception method of claim 2, wherein, The projection of the image points in the three-dimensional space to the fisheye camera imaging plane according to the imaging principle of the fisheye camera comprises: the image points are converted into unit vectors on a unit sphere; a radial distance relationship between a light incidence angle and a projection point of a center point of the fisheye camera imaging plane to the fisheye camera imaging plane is calculated according to the imaging principle of the fisheye camera; projection coordinates of the image points on the fisheye camera imaging plane are calculated according to the unit vectors and the radial distance relationship.

4. The fisheye camera based panoramic perception method of claim 3, wherein, The method further comprises: internal parameters of the fisheye camera are calibrated by using least square optimization and according to polynomial parameters of the projection coordinates, an affine transformation matrix of the fisheye camera, a position coordinate of the center point of the fisheye camera imaging plane when the light incidence angle is 0, and a rotation matrix and a translation matrix between the fisheye camera and a calibration board.

5. The fisheye camera based panoramic perception method of claim 1, wherein, the matching cost calculation on the left image features and the right image features corresponding to each binocular camera comprises: Hamming distance is calculated for the left image feature and the right image feature corresponding to each of the binocular cameras as a matching cost calculation.

6. The fisheye camera based panoramic perception method of claim 1, wherein, Before the four depth maps covering the panorama of the positions of the four fisheye cameras are determined according to the disparity map and the parameters of the virtual pinhole camera, the method further comprises: At least one of disparity smoothing, disparity interpolation, disparity consistency checking or edge preservation is performed on the disparity map to obtain a disparity-optimized disparity map. The four depth maps covering the panorama of the positions of the four fisheye cameras are determined according to the disparity-optimized disparity map. The four depth maps covering the panorama of the positions of the four fisheye cameras are determined according to the disparity-optimized disparity map.

7. A fisheye camera based panoramic perception device, comprising: It comprises: A fisheye image acquisition unit is configured to acquire four fisheye images corresponding to four fisheye cameras; wherein the four fisheye cameras are respectively placed at the four vertices of a square, and the lenses are directed outward from the diagonal of the square; each fisheye camera generates two virtual pinhole cameras, and each virtual pinhole camera constitutes a binocular camera with a neighboring fisheye camera; A fisheye image conversion unit is configured to convert each fisheye image into a left image and a right image corresponding to the binocular camera according to a mapping table generated in advance by the imaging principle of the fisheye camera and the parameters of the virtual pinhole camera; An image feature extraction unit is configured to extract left image features corresponding to the left image and right image features corresponding to the right image of each binocular camera using a binary neural network; A disparity calculation unit is configured to perform disparity calculation on the left image features and the right image features corresponding to each binocular camera to obtain four depth maps covering the panorama of the positions of the four fisheye cameras. The disparity calculation on the left image features and the right image features corresponding to each binocular camera to obtain four depth maps covering the panorama of the positions of the four fisheye cameras comprises: Matching cost calculation is performed on the left image features and the right image features corresponding to each binocular camera to determine the right image pixel corresponding to the left image pixel of the left image feature in the right image feature; All right image pixels corresponding to each left image pixel are smoothed to aggregate a global matching cost; A semi-global matching algorithm is used to calculate the pixel displacement of the right image pixel corresponding to each left image pixel in the global matching cost to obtain a disparity value, and a corresponding disparity map is determined according to the disparity value. A four depth maps covering the panorama of the positions of the four fisheye cameras are determined according to the disparity map and the parameters of the virtual pinhole camera.

8. An electronic device, comprising: It comprises a processor and a memory; The memory is configured to store a program; The processor executes the program to implement a fisheye camera-based panoramic perception method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement a fisheye camera-based panoramic perception method according to any one of claims 1 to 6.

Images