A monocular ranging method based on an in-vehicle fisheye camera

Abstract

The present invention provides a monocular ranging method based on an in-vehicle fish-eye camera, which can improve the inference speed, has better real-time performance, and is applicable to more application scenarios; a lightweight ranging model is constructed, and the ranging model includes: a backbone network and a head module connected in sequence, and each convolution calculation in the backbone network and the head module is implemented based on sparse deformable convolution; during the process of extracting features by the backbone network, instead of calculating and outputting for each pixel point separately, multiple pixel points participating in the calculation are output as adjacent local N feature points; when the head module realizes target detection and depth prediction based on the features output by the backbone network, instead of inferring the depth information of each pixel point separately, it only needs to calculate the distance corresponding to the predicted target to obtain the distance value corresponding to its original feature point.

Description

Technical Field

[0001] The present invention relates to the technical field of monocular ranging, and specifically to a monocular ranging method based on an in-vehicle fisheye camera. Background Art

[0002] Combining a distance sensor with a video sensor for ranging, speed measurement, and target recognition is the main data acquisition and analysis method for current intelligent vehicles and unmanned vehicles. Among them, monocular ranging has gradually become one of the current mainstream research directions because it only requires a monocular in-vehicle camera, does not require complex registration and synchronization, has a simple measurement principle, and makes full use of data.

[0003] The core imaging model of monocular ranging is the pinhole imaging model. In pinhole imaging, light travels in a straight line. When the light emitted or reflected by an object passes through the pinhole, an inverted real image will be formed on the imaging plane on the other side of the pinhole. Assume the actual height of the object is H, its imaging height in the image is h, the camera focal length is f, and the actual distance between the object and the camera is d. According to the principle of similar triangles, H / h = d / f can be obtained. In practical applications, an imaging picture of the measurement target is obtained through an in-vehicle monocular camera. When the camera focal length f of the in-vehicle monocular camera and the actual height H of the object are known, the height h of the object imaging in the image is measured, and then the horizontal distance d between the measurement target and the monocular camera can be calculated. During the calculation process, conversions between the camera coordinate system, the image coordinate system, the world coordinate system, and the pixel coordinate system are involved. At the same time, factors such as the image distance error of the monocular camera also need to be considered. Based on the image distance error formula of the monocular camera, the object distance is compensated through the incident error formula to obtain the true actual distance. Therefore, the actual calculation process is very complex and the calculation real-time performance is poor.

[0004] With the wide application of machine learning and deep learning, in the prior art, some technicians also use neural networks to learn the complex mapping relationship between image features and object distances. For example, during the training process, images with accurate distance annotations are used to construct a training dataset to train the neural network. The neural network gradually learns to predict the distance of an object based on various features in the image, such as the shape, texture, color, etc. of the object by continuously adjusting its own parameters. After the training is completed, the image to be ranged is input into the trained neural network model, and the model can output the corresponding pixel or the distance value of the target. This method can automatically learn complex feature representations, avoiding the limitations of manual feature extraction, and showing good ranging performance in some complex scenarios. However, since the monocular vision ranging method based on machine learning and deep learning requires a large amount of data for learning, and the generalization ability of the obtained model is restricted by many factors, in the process of in-vehicle monocular vision ranging, under the current technical conditions, the in-vehicle monocular vision distance measurement method has technical problems such as low measurement accuracy and real-time performance, complex measurement process, and poor generalization ability.

[0005] In the prior art, most monocular vehicle-mounted camera ranging methods are mostly based on non-fisheye cameras, and the ranging of fisheye cameras is inaccurate; moreover, the internal parameters of the cameras equipped on different vehicles are different, resulting in obvious differences in the ranging of targets in the same scene; at the same time, due to the relatively high driving speed of the vehicle, the ranging of the vehicle-mounted camera requires the network to achieve real-time effects. However, most monocular ranging methods have a slow inference speed because they need to predict the depth of each pixel point, and are not suitable for application scenarios with high real-time requirements. Summary of the Invention

[0006] In order to solve the problem that the real-time performance of the monocular ranging method in the prior art is poor and cannot meet all application scenarios, the present invention provides a monocular ranging method based on a vehicle-mounted fisheye camera, which can improve the inference speed, has better real-time performance, and is applicable to more application scenarios.

[0007] The technical solution of the present invention is as follows: A monocular ranging method based on a vehicle-mounted fisheye camera, characterized in that it includes the following steps:

[0008] S1: Collect images respectively based on a monocular fisheye camera to obtain an image to be processed;

[0009] Perform distortion correction on the image to be processed to obtain the corrected image to be processed;

[0010] S2: Label the corrected image to be processed, label all preset targets included in the picture and their actual distance d relative to the monocular fisheye camera to obtain the labeled image to be processed;

[0011] S3: Construct a training set and a validation set based on the labeled image to be processed;

[0012] S4: Construct a lightweight ranging model;

[0013] The ranging model includes: a backbone network and a head module connected in sequence;

[0014] The backbone network realizes the extraction of image features of the input image;

[0015] The convolution method in the backbone network is set to sparse deformable convolution, and the expression of the sparse deformable convolution is:

[0016] y i =wx + b i ;

[0017] where y iis the i-th feature point in the local area of each convolution output of the current layer, where i < N, N is the total number of feature points output by each convolution calculation of the sparse deformable convolution, and N > 1;

[0018] x is the output of the previous layer of the network, w is the network weight of the current layer, and b is the bias term corresponding to the i-th feature point;

[0019] The head module processes the Feature extracted by the backbone to complete object detection and depth prediction; each convolution method in the head module is set to sparse deformable convolution;

[0020] The detected object is denoted as the predicted object, and the output of the head module includes: the position of each predicted object in the input image, the category of the predicted object, and the distance between the predicted object and the camera;

[0021] S6: Train the ranging model based on the training data set to obtain the trained ranging model;

[0022] During the training process, the head module uses the Loss function for the loss function of distance prediction; dis function;

[0023] ;

[0024] where d pre represents the distance predicted by the network, d gt represents the real distance, d max represents the maximum distance in the training data, and a is a hyperparameter;

[0025] S7: Collect real-time images through the on-vehicle monocular fisheye camera on the vehicle to be ranged, denoted as: the image to be detected;

[0026] After performing distortion correction on the image to be detected, the corrected image to be detected is obtained;

[0027] S8: Send the corrected image to be detected into the trained ranging model in real time to obtain the predicted objects in the image to be detected, as well as the position of each predicted object in the input image, the category of the predicted object, and the distance between the predicted object and the camera.

[0028] Its further feature lies in:

[0029] It further includes the following steps:

[0030] S9: Obtain the position of each predicted object in the input image and the distance between the predicted object and the camera, and convert the image coordinates to the world coordinate system according to the internal and external parameters of the camera, that is, obtain the orientation of the predicted object relative to the vehicle to be ranged;

[0031] It also includes a ranging method for on-vehicle cameras that can be compatible with different parameters, specifically including the following steps:

[0032] In step S1, images are collected respectively based on monocular fisheye cameras with multiple different parameters to obtain images to be processed;

[0033] Before implementing step S3, the following steps are also required:

[0034] Map the images captured by different cameras to the same focal length space according to the focal length f, and correct the marked d using the following calculation method:

[0035] d * = (d*f) / 1000;

[0036] where d * is the true value of the target distance in the unified focal length space;

[0037] Use the value of d * to mark all preset targets included in the picture and their actual distances from the monocular fisheye camera, and obtain the marked image to be processed;

[0038] The backbone network is constructed based on the CSPDarknet model, and the convolution method in the CSPDarknet model is modified to the sparse deformable convolution;

[0039] In the sparse deformable convolution, the number N of local feature points output by each convolution takes the value of 4, i ∈ (0,1,2,3);

[0040] The structure of the head module includes: three parallel prediction channels;

[0041] The three prediction channels are respectively: the position prediction channel of the prediction target in the image, the target category prediction channel, and the distance prediction channel between the prediction target and the camera;

[0042] Each prediction channel includes: a ConvModule and a Conv2d connected in sequence;

[0043] In step S1, the image to be processed is compensated for distortion based on the homography matrix and affine transformation.

[0044] A monocular ranging method based on an in-vehicle fisheye camera provided by this application constructs a lightweight ranging model. The ranging model includes a backbone network and a head module connected in sequence. Each convolution calculation in the backbone network and the head module is implemented based on sparse deformable convolution. Since the target distance prediction mainly relies on the depth information in the images collected by the in-vehicle camera, and when predicting the depth of the target, the depth differences of all pixel points in the area where the target is located are relatively small, sparse deformable convolution is designed for this feature of the depth information in this application. In the feature map obtained after each convolution calculation, the values of adjacent N points are relatively close. The N points share the local feature points when convolving with x, except for the bias b which is different. During the process of the backbone network extracting features, it does not need to calculate and output for each pixel point separately, but outputs multiple pixel points participating in the calculation as adjacent local N feature points. When the head module performs target detection and depth prediction based on the features output by the backbone network, it does not need to reason about the depth information of each pixel point separately, but only needs to calculate the distance corresponding to the predicted target to obtain the distance value corresponding to its original feature point. During the process of the ranging model extracting features and making predictions on the input image, the calculation is completed based on sparse deformable convolution, effectively reducing the amount of convolution calculation, improving the inference calculation speed, and ensuring that this method has better real-time performance. At the same time, because the target distance prediction mainly relies on the depth information in the images collected by the in-vehicle camera, this application also designs a Loss function for the head module responsible for target detection and depth prediction based on the relationship between the target and the distance value of the camera, effectively improving the convergence speed in model training and shortening the model training implementation time. dis Function, effectively improving the convergence speed in model training and shortening the model training implementation time. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the ranging model structure;

[0046] Figure 2 It is a schematic diagram of the principle of sparse deformable convolution;

[0047] Figure 3 It is a schematic diagram of the head module structure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] As Figures 1 to 3 shown, this application includes a monocular ranging method based on an in-vehicle fisheye camera, which includes the following steps.

[0049] S1: Collect images separately based on a monocular fisheye camera to obtain the images to be processed. Since the images collected by the fisheye camera are all distorted to some extent, in order to improve the accuracy of subsequent object detection, it is necessary to perform distortion correction on the images to be processed based on distortion correction methods in the existing technology, such as: checkerboard calibration method, horizontal expansion method, longitude and latitude method, etc., to obtain the corrected images to be processed. In this embodiment, the homography matrix and affine transformation are used to compensate for the distortion.

[0050] If only building a ranging model for a specific monocular fisheye camera, it is only necessary to collect the images taken by this camera to construct a training dataset. However, if expecting to build a general ranging model, in order to be compatible with vehicle-mounted cameras with different parameters, when constructing the training set, it is necessary to consider the differences in parameters of various existing vehicle-mounted monocular fisheye cameras, and it is necessary to collect images separately based on multiple monocular fisheye cameras with different parameters to obtain the images to be processed.

[0051] S2: Label the corrected images to be processed, label all the preset targets included in the picture and their actual distance d relative to the monocular fisheye camera, to obtain the labeled images to be processed.

[0052] If it is necessary to build a general one compatible with various different models of monocular fisheye cameras, the following steps also need to be carried out when constructing the training dataset:

[0053] Map the images taken by different cameras to the same focal length space according to the focal length f, and use the following calculation method to correct the labeled d:

[0054] d * = (d*f) / 1000;

[0055] where, d * is the true value of the target distance in the unified focal length space;

[0056] Use the value of d * to label all the preset targets included in the picture and their actual distance corresponding to the monocular fisheye camera, to obtain the labeled images to be processed; the ranging model trained using this true value of the target distance d * in the unified focal length space is applicable to vehicle-mounted cameras with any parameters.

[0057] S3: Based on the labeled images to be processed, construct a training set and a validation set. The specific construction methods of the training set and the validation set are implemented based on the existing technology.

[0058] S4: Build a lightweight ranging model.

[0059] Since the target distance prediction mainly relies on the depth information in the images collected by the vehicle-mounted camera, and when predicting the depth of the target, the depth difference of all pixel points in the area where the target is located is small, this application designs a sparse deformable convolution for this feature of the depth information. The sparse deformable convolution can greatly reduce the computational complexity of the convolution.

[0060] The expression of the sparse deformable convolution is:

[0061] y i = wx + b i ;

[0062] Where y i is the i-th feature point in the local area of each convolution output of the current layer, i < N, N is the total number of feature points output by each convolution calculation of the sparse deformable convolution, N > 1; x is the output of the previous layer of the network, w is the network weight of the current layer, and b is the bias term corresponding to the i-th feature point.

[0063] As Figure 2 shown, it is a schematic diagram of the principle of the sparse deformable convolution. During convolution, the output x of the previous layer is used as the input of this layer, and the first layer of convolution uses the image input by the input layer as the input of this layer.

[0064] During the calculation of the sparse deformable convolution, M feature points are randomly obtained from the input x each time. The specific number M of the feature points of x obtained is determined according to the number of specific convolution kernels. These randomly obtained M feature points are subjected to convolution calculation and become adjacent N feature points as the output. These N output feature points share the local feature points when convolving x.

[0065] The deformable convolution adds an offset learning in a convolution layer. Through the training of the training set, the network weight w of the current layer in the deformable convolution and the bias term b corresponding to the i-th feature point can be fully learned.

[0066] For the specific value of N, the accuracy and computational efficiency need to be considered simultaneously; when N takes a larger value, the accuracy can be improved, but the computational complexity will increase; when N takes a smaller value, the computational complexity can be reduced, but the accuracy will be lost; so it is necessary to balance the specific computational accuracy requirements and real-time requirements. At the same time, to ensure the improvement of computational efficiency, the value of N also needs to meet the condition: N < M.

[0067] In this embodiment, for the sparse deformable convolution applied in the ranging model, the number N of local feature points output by each convolution is 4, i ∈ (0, 1, 2, 3), which can meet the requirements of both computational efficiency and ranging accuracy.

[0068] As Figure 1 shown, the ranging model in this application includes: a backbone network and a head module connected in sequence.

[0069] The backbone network extracts the image features of the input image; the convolution method in the backbone network is set to sparse deformable convolution.

[0070] Specifically, the backbone network can be implemented based on the commonly used backbone network models in the prior art, such as: VGG, Resnet, DeseNet, etc. In this embodiment, the backbone network is constructed based on the CSPDarknet model, and the convolution method in the CSPDarknet model is modified to sparse deformable convolution, which effectively improves the calculation efficiency, reduces the calculation amount, and at the same time can ensure the detection accuracy.

[0071] The head module processes the Feature extracted by the backbone to complete object detection and depth prediction; each convolution method in the head module is set to sparse deformable convolution;

[0072] The detected object is denoted as the predicted object, and the output of the head module includes: the position of each predicted object in the input image, the category of the predicted object, and the distance between the predicted object and the camera.

[0073] Such as Figure 3 As shown, the structure of the head module in this application includes: three parallel prediction channels;

[0074] The three prediction channels are respectively: the position prediction channel bbox pred of the predicted object in the image, the object category prediction channel class pred, and the distance prediction channel distance pred between the predicted object and the camera. Each prediction channel includes: a ConvModule and a Conv2d connected in sequence.

[0075] Among them, ConvModule is the basic module for performing convolution operations in the deep learning framework, mainly used to extract and process image features to achieve the detection and recognition of objects. Specifically, it includes a convolutional layer (Conv), batch normalization (BN), and an activation function. conv2d is a 2D convolution.

[0076] S6: Train the ranging model based on the training dataset to obtain a trained ranging model;

[0077] During the training process, the loss function for distance prediction in the head module uses the Loss dis function;

[0078] ;

[0079] Among them, d preRepresents the distance predicted by the network, d gt Represents the actual distance, d max Represents the maximum distance in the training data. a is a hyperparameter obtained through training.

[0080] During the neural network training process, the loss function is used to evaluate the difference or error between the predicted value and the true label. Generally, the smaller the loss function value, the closer the predicted result is to the true value, and the better the model performance. The ranging model in this application is mainly used to predict the distance between the target and the camera, and at the same time will output the position and category of the target. Therefore, in this application, the distance between the target and the camera is used as an indicator to measure the training effect of the model. According to d pre 、d gt 、d max Three distance values are used to construct the loss function. When d pre ≥d max , the loss function is not calculated. And the closer d pre and d gt are, the better the training effect, effectively improving the training convergence speed of the model.

[0081] The specific construction details of the ranging model, as well as the training method and process, are implemented based on the existing technology.

[0082] The model trained based on this method is applicable to on-vehicle cameras with different parameters, can not only obtain the distance of the target but also detect the position of the target relative to the vehicle itself. At the same time, the lightweight design enables the model to achieve a good real-time effect.

[0083] S7: Collect real-time images through the on-vehicle monocular fisheye camera on the vehicle to be ranged, denoted as: the image to be detected;

[0084] After correcting the distortion of the image to be detected, the corrected image to be detected is obtained.

[0085] S8: Send the corrected image to be detected into the trained ranging model in real time to obtain the predicted targets in the image to be detected, as well as the position of each predicted target in the input image, the category of the predicted target, and the distance between the predicted target and the camera.

[0086] S9: Obtain the position of each prediction target in the input image and the distance between the prediction target and the camera. Convert the image coordinates to the world coordinate system according to the internal and external parameters of the camera, that is, obtain the orientation of the prediction target relative to the vehicle to be distance-measured. According to the results output by the distance measurement model in this application, the relative position between the prediction target and the vehicle where the camera is located can be calculated. The internal parameters of the vehicle-mounted camera are constants, which have been calibrated and stored in the vehicle before the vehicle goes on the line. The external parameters are the pose of the vehicle-mounted camera in the world coordinate system, which describes the relative position relationship between the camera coordinates and the world coordinate system. After obtaining the relative position relationship between the camera head coordinates and the world coordinate system, the relative orientation relationship between the prediction target and the vehicle where the camera is located can be obtained by using the distance between the prediction target and the camera. The specific calculation details such as the conversion process of the internal and external parameters can be implemented based on existing technologies such as calibration algorithms and three-dimensional positioning methods.

[0087] After using the technical solution of the present invention, by unifying the distances of the targets in the images captured by cameras with different parameters into the same focal length space, the trained model is applicable to vehicle-mounted cameras with different parameters; a lightweight network structure is designed based on sparse deformable convolution, so that the model inference speed reaches real-time effect and reduces the influence of too fast vehicle speed on target distance measurement; the designed network with target detection and distance measurement functions can not only obtain the distance of the target, but also determine the position of the target relative to the vehicle itself according to the target detection result and camera parameters.

Claims

1. A monocular ranging method based on an in-vehicle fisheye camera, characterized in that, It includes the following steps: S1: Collect images respectively based on a monocular fisheye camera to obtain the images to be processed; Perform distortion correction on the images to be processed to obtain the corrected images to be processed; S2: Label the corrected images to be processed, label all preset targets included in the images and their actual distances d relative to the monocular fisheye camera, to obtain the labeled images to be processed; S3: Based on the labeled images to be processed, construct a training set and a validation set; S4: Construct a lightweight ranging model; The ranging model includes: a backbone network and a head module connected in sequence; The backbone network realizes the extraction of image features of the input image; the head module processes the features extracted by the backbone to complete object detection and depth prediction; The convolution methods in both the backbone network and the head module are set to sparse deformable convolutions; The detected targets are denoted as predicted targets, and the output of the head module includes: the position of each predicted target in the input image, the category of the predicted target, and the distance between the predicted target and the camera; The expression of the sparse deformable convolution is: y i = wx + b i ; Among them, y i is the i-th feature point of the local area of each convolution output in the current layer, where i < N, N is the total number of feature points output by each convolution calculation of the sparse deformable convolution, and N > 1; x is the output of the upper layer of the network, w is the network weight of the current layer, and b i is the bias term corresponding to the i-th feature point; In the feature map obtained after each convolution calculation, the values of adjacent N points are relatively close. The N points share the local feature points when convolving with x, except for the bias b i which is different; When performing sparse deformable convolution calculation, randomly obtain M feature points in the input x each time. The specific number M of the feature points of x obtained is determined according to the number of specific convolution kernels. These randomly obtained M feature points are subjected to convolution calculation to become adjacent N feature points as the output; In the sparse deformable convolution, the number N of local feature points output by each convolution is 4, i ∈ (0, 1, 2, 3); The backbone network is constructed based on the CSPDarknet model, and the convolution method in the CSPDarknet model is modified to the sparse deformable convolution; The structure of the head module includes: three parallel prediction channels; The three prediction channels are respectively: the position prediction channel of the predicted target in the image, the target category prediction channel, and the distance prediction channel between the predicted target and the camera; Each prediction channel includes: a ConvModule and a Conv2d connected in sequence; During the feature extraction process of the backbone network, it is not necessary to calculate and output for each pixel point respectively. Instead, multiple pixel points participating in the calculation are output as adjacent local N feature points; when the head module realizes object detection and depth prediction based on the features output by the backbone network, it is not necessary to reason about the depth information of each pixel point respectively. Instead, it only needs to calculate the distance corresponding to the predicted target to obtain the distance value corresponding to its original feature point; S6: Train the ranging model based on the training set to obtain the trained ranging model; In step S6, during the training process, the head module uses the Loss dis function for the loss function of distance prediction; ； Among them, d pre represents the distance predicted by the network, and d gt represents the true distance, and d max represents the maximum distance in the training data, where a is a hyperparameter; S7: Collect real-time images through the on-vehicle monocular fisheye camera on the vehicle to be ranged, denoted as: the images to be detected; After performing distortion correction on the images to be detected, obtain the corrected images to be detected; S8: Real-time send the corrected image to be detected into the trained ranging model, to obtain the targets predicted in the image to be detected, as well as the positions of each predicted target in the input image, the categories of the predicted targets, and the distances between the predicted targets and the camera.

2. The monocular ranging method based on an in-vehicle fisheye camera according to claim 1, wherein: It further includes the following steps: S9: Obtain the positions of each predicted target in the input image and the distances between the predicted targets and the camera, and convert the image coordinates to the world coordinate system according to the internal and external parameters of the camera, that is, obtain the azimuths of the predicted targets relative to the vehicle to be ranged.

3. The monocular ranging method based on an in-vehicle fisheye camera according to claim 1, wherein: It further includes a ranging method for implementing compatibility with vehicle-mounted cameras with different parameters, specifically including the following steps: In step S1, images are respectively collected based on multiple monocular fisheye cameras with different parameters to obtain images to be processed; Before implementing step S3, the following steps are also required: Map the images captured by different cameras to the same focal length space according to the focal length f, and correct the labeled d using the following calculation method: d * = (d*f) / 1000; where d * is the true value of the target distance in the unified focal length space; Use d * to mark all the preset targets included in the picture and their actual distances from the corresponding monocular fisheye camera, and obtain the to-be-processed image after annotation.

4. The monocular ranging method based on an in-vehicle fisheye camera according to claim 1, wherein: In step S1, the images to be processed are compensated for distortion based on the homography matrix and affine transformation.

Images