Monocular distance measurement method based on vehicle-mounted fisheye camera

Abstract

The invention provides a monocular distance measurement method based on a vehicle-mounted fisheye camera, which can improve the reasoning speed, has better real-time performance and is suitable for more application scenes. A lightweight distance measurement model is constructed, the distance measurement model comprises a backbone network and a head module which are connected in sequence, and each convolution calculation in the backbone network and the head module is realized based on sparse deformable convolution; in the feature extraction process of the backbone network, the backbone network does not need to calculate and output each pixel point, but outputs a plurality of pixel points participating in calculation as adjacent local N feature points; when the head module realizes target detection and depth prediction based on the features output by the backbone network, the depth information of each pixel point does not need to be reasoned separately, and the distance value corresponding to the original feature point of the target can be obtained only by calculating the distance corresponding to the predicted target.

Description

Technical Field

[0001] The invention relates to the technical field of monocular distance measurement, and in particular to a monocular distance measurement method based on a vehicle-mounted fisheye camera. Background Art

[0002] The combination of distance sensors and video sensors for distance measurement, speed measurement and target recognition is currently the main means of data collection and analysis for smart cars and unmanned vehicles. Among them, monocular distance measurement has gradually become one of the current mainstream research directions because it only needs to use a monocular vehicle-mounted camera, does not require complex registration and synchronization, has a simple measurement principle, and fully utilizes data.

[0003] The core imaging model of monocular ranging is the pinhole imaging model. In pinhole imaging, light propagates in a straight line. When the light emitted or reflected by the object passes through the pinhole, an inverted real image is formed on the imaging plane on the other side of the pinhole. Assume that the actual height of the object is H, its imaging height in the image is h, the focal length of the camera is f, and the actual distance between the object and the camera is d. According to the principle of similar triangles, H / h=d / f can be obtained. In practical applications, the imaging picture of the measurement target is obtained by the on-board monocular camera. When the camera focal length f of the on-board monocular camera and the actual height H of the object are known, the height h of the object imaging in the image is measured, and the horizontal distance d between the measurement target and the monocular camera can be calculated. In the calculation process, the conversion between the camera coordinate system, the image coordinate system, the world coordinate system and the pixel coordinate system is involved. At the same time, factors such as the image distance error of the monocular camera must also be considered. Based on the image distance error formula of the monocular camera, the object distance is compensated by the incident error formula to obtain the real actual distance. Therefore, the actual calculation process is very complicated and the real-time calculation is poor.

[0004] With the widespread application of machine learning and deep learning, in the prior art, some technicians also use neural networks to learn the complex mapping relationship between image features and object distances. For example, in the training process, images with accurate distance annotations are used to construct a training data set to train the neural network. The neural network gradually learns to predict the distance of the object based on various features in the image, such as the shape, texture, color, etc. of the object by continuously adjusting its own parameters. After the training is completed, the image to be measured is input into the trained neural network model, and the model can output the corresponding pixel or target distance value. This method can automatically learn complex feature representations, avoid the limitations of manual feature extraction, and show good ranging performance in some complex scenes. However, since the monocular vision ranging method based on machine learning and deep learning requires a large amount of data for learning, and the generalization ability of the obtained model is restricted by many factors, in the process of on-board monocular vision ranging, under current technical conditions, the on-board monocular vision distance measurement method has technical problems such as low measurement accuracy and real-time performance, complex measurement process, and poor generalization ability.

[0005] In the prior art, most of the monocular vehicle-mounted camera ranging methods are based on non-fisheye cameras, and the ranging of fisheye cameras is inaccurate; moreover, the internal parameters of the cameras equipped on different vehicles are different, resulting in obvious ranging differences for the same target in the same scene; at the same time, due to the relatively fast driving speed of the vehicle, the ranging of the vehicle-mounted camera requires the network to achieve real-time effects. However, most of the monocular ranging methods have slow inference speeds because they need to predict the depth of each pixel point, and are not suitable for application scenarios with high real-time requirements. Summary of the Invention

[0006] In order to solve the problem that the real-time performance of the monocular ranging method in the prior art is poor and cannot meet all application scenarios, the present invention provides a monocular ranging method based on a vehicle-mounted fisheye camera, which can improve the inference speed, has better real-time performance, and is applicable to more application scenarios.

[0007] The technical solution of the present invention is as follows: A monocular ranging method based on a vehicle-mounted fisheye camera, characterized in that it includes the following steps: S1: Collect images respectively based on a monocular fisheye camera to obtain images to be processed; Perform distortion correction on the images to be processed to obtain the corrected images to be processed; S2: Label the corrected images to be processed, label all preset targets included in the pictures and their actual distances d relative to the monocular fisheye camera to obtain the labeled images to be processed; S3: Construct a training set and a validation set based on the labeled images to be processed; S4: Construct a lightweight ranging model; The ranging model includes: a backbone network and a head module connected in sequence; The backbone network realizes the extraction of image features of the input image; The convolution method in the backbone network is set to sparse deformable convolution, and the expression of the sparse deformable convolution is: y i =wx + b i ; where y i is the i-th feature point in the local area of each convolution output of the current layer, i < N, N is the total number of feature points output by each convolution calculation of the sparse deformable convolution, N > 1; x is the output of the previous layer of the network, w is the network weight of the current layer, and b is the bias term corresponding to the i-th feature point; The head module processes the features extracted by the backbone to complete target detection and depth prediction; each convolution mode in the head module is set to sparse deformable convolution; The detected target is recorded as a predicted target, and the output of the head module includes: the position of each predicted target in the input image, the category of the predicted target, and the distance between the predicted target and the camera; S6: Training the distance measurement model based on the training data set to obtain the trained distance measurement model; During the training process, the head module uses Loss as the loss function for distance prediction. dis function; ; Among them, d pre represents the distance predicted by the network, d gt Indicates the true distance, d max Represents the maximum distance in the training data, and a is a hyperparameter; S7: collect real-time images through the on-board monocular fisheye camera on the vehicle to be measured, which is recorded as: image to be detected; After performing distortion correction on the image to be detected, a corrected image to be detected is obtained; S8: Send the corrected image to be detected to the trained distance measurement model in real time to obtain the predicted targets in the image to be detected, as well as the position of each predicted target in the input image, the category of the predicted target, and the distance between the predicted target and the camera.

[0008] It is further characterized by: It also includes the following steps: S9: Obtain the position of each predicted target in the input image and the distance between the predicted target and the camera, and convert the image coordinates into a world coordinate system according to the internal and external parameters of the camera, that is, obtain the orientation of the predicted target relative to the vehicle to be measured; It also includes a distance measurement method for realizing a vehicle-mounted camera compatible with different parameters, which specifically includes the following steps: In step S1, images are collected respectively based on a monocular fisheye camera with a plurality of different parameters to obtain images to be processed; Before the implementation of step S3, the following steps are required: According to the focal length f, the images captured by different cameras are mapped to the same focal length space, and the marked d is corrected using the following calculation method: d * = (d*f) / 1000; Among them, d * is the true value of the target distance in the unified focal length space; Use d * The value of is used to mark the actual distances between all preset targets in the image and their corresponding monocular fisheye cameras, and the marked image to be processed is obtained; The backbone network is constructed based on the CSPDarknet model, and the convolution mode in the CSPDarknet model is modified to the sparse deformable convolution; In the sparse deformable convolution, the number N of local feature points output by each convolution is 4, i∈(0,1,2,3); The structure of the head module includes: three parallel prediction channels; The three prediction channels are: a channel for predicting the position of a target in an image, a channel for predicting the target category, and a channel for predicting the distance between a target and a camera; Each of the prediction channels comprises: a ConvModule and a Conv2d connected in sequence; In step S1, the distortion of the image to be processed is compensated based on the homography matrix and the affine transformation.

[0009] The present application provides a monocular ranging method based on a vehicle-mounted fisheye camera, which constructs a lightweight ranging model. The ranging model includes: a backbone network and a head module connected in sequence, in which each convolution calculation is implemented based on sparse deformable convolution; because the target distance prediction mainly relies on the depth information in the image collected by the vehicle-mounted camera, and when predicting the depth of the target, the depth difference of all pixels in the target area is small, so the present application designs sparse deformable convolution for this feature of depth information, and in the feature map obtained after each convolution calculation, the values ​​of adjacent N points are close, and the N points share the local feature points during the x-convolution. , only the bias b is different; in the process of extracting features, the backbone network does not need to calculate and output each pixel separately, but outputs multiple pixels involved in the calculation as adjacent local N feature points; when the head module realizes target detection and depth prediction based on the features output by the backbone network, it does not need to infer the depth information of each pixel separately, but only needs to calculate the distance corresponding to the predicted target to obtain the distance value corresponding to its original feature point; in the process of feature extraction and prediction of the input image by the ranging model, the calculation is completed based on sparse deformable convolution, which effectively reduces the amount of convolution calculation, improves the speed of inference calculation, and ensures that the method has better real-time performance. At the same time, because the target distance prediction mainly relies on the depth information in the image collected by the on-board camera, this application also designs Loss for the head module responsible for target detection and depth prediction based on the relationship between the distance value of the target and the camera. disFunction, which effectively improves the convergence speed in model training and shortens the model training implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram of the distance measurement model structure; Figure 2 Schematic diagram of the principle of sparse deformable convolution; Figure 3 This is a schematic diagram of the head module structure. DETAILED DESCRIPTION

[0011] like Figure 1~Figure 3 As shown, the present application includes a monocular ranging method based on a vehicle-mounted fisheye camera, which includes the following steps.

[0012] S1: Based on the monocular fisheye camera, images are collected respectively to obtain images to be processed; because the images collected by the fisheye camera are deformed to a certain extent, in order to improve the accuracy of subsequent target detection, it is necessary to perform distortion correction on the images to be processed based on the distortion correction methods in the prior art, such as chessboard calibration method, lateral expansion method, longitude and latitude method, etc., to obtain the corrected images to be processed. In this embodiment, the distortion is compensated based on the homography matrix and affine transformation.

[0013] If you only want to build a ranging model for a specific monocular fisheye camera, you only need to collect images taken by this camera to build a training data set. If you want to build a universal ranging model, in order to achieve compatibility with vehicle-mounted cameras with different parameters, you need to consider the differences in parameters of various existing vehicle-mounted monocular fisheye cameras when building the training set. You need to collect images based on monocular fisheye cameras with different parameters to obtain images to be processed.

[0014] S2: Annotate the corrected image to be processed, annotate all preset targets included in the image and their actual distances d relative to the monocular fisheye camera, and obtain the annotated image to be processed.

[0015] If you need to build a universal monocular fisheye camera that is compatible with various models, you need to perform the following steps when building a training dataset: According to the focal length f, the images captured by different cameras are mapped to the same focal length space, and the marked d is corrected using the following calculation method: d * = (d*f) / 1000; Among them, d * is the true value of the target distance in the unified focal length space; Use d *The values mark all the preset targets included in the image and the actual distances of their corresponding monocular fisheye cameras, obtaining the to-be-processed image after annotation; using the true value d of the target distance in the unified focal length space * The trained ranging model is applicable to vehicle-mounted cameras with any parameters.

[0016] S3: Based on the to-be-processed image after annotation, construct a training set and a validation set. The specific construction methods of the training set and the validation set are implemented based on the prior art.

[0017] S4: Construct a lightweight ranging model.

[0018] Since the target distance prediction mainly relies on the depth information in the images collected by the vehicle-mounted camera, and when predicting the depth of the target, the depth differences of all pixel points in the area where the target is located are relatively small, so in this application, a sparse deformable convolution is designed for this feature of the depth information. The computational amount of the convolution can be greatly reduced through the sparse deformable convolution.

[0019] The expression of the sparse deformable convolution is: y i = wx + b i ; where, y i is the i-th feature point of the local area output by each convolution of the current layer, i < N, N is the total number of feature points output by each convolution calculation of the sparse deformable convolution, N > 1; x is the output of the previous layer of the network, w is the network weight of the current layer, and b is the bias term corresponding to the i-th feature point.

[0020] As Figure 2 shown, it is a schematic diagram of the principle of the sparse deformable convolution. During convolution, the output x of the previous layer is used as the input of this layer, and the first layer of convolution uses the image input by the input layer as the input of this layer.

[0021] During the calculation of the sparse deformable convolution, M feature points are randomly obtained from the input x each time. The specific number M of the obtained feature points of x is determined according to the number of specific convolution kernels. These randomly obtained M feature points are subjected to convolution calculation and become adjacent N feature points as the output. These N output feature points share the local feature points when convolving x.

[0022] The deformable convolution adds an offset learning in a convolution layer. Through the training of the training set, the network weight w of the current layer in the deformable convolution and the bias term b corresponding to the i-th feature point can be fully learned.

[0023] The specific value of N needs to consider both accuracy and computational efficiency. A larger value of N can improve accuracy, but the amount of computation will increase. A smaller value of N can reduce the amount of computation, but the accuracy will be lost. Therefore, it is necessary to balance the specific computational accuracy requirements and real-time requirements. At the same time, in order to ensure the improvement of computational efficiency, the value of N also needs to meet the following conditions: <M。

[0024] In this embodiment, for the sparse deformable convolution used in the ranging model, the number N of local feature points output by each convolution is 4, i∈(0,1,2,3), which can meet the requirements of computational efficiency and ranging accuracy at the same time.

[0025] like Figure 1 As shown, the ranging model in the present application includes: a backbone network and a head module connected in sequence.

[0026] The backbone network realizes the extraction of image features of the input image; the convolution mode in the backbone network is set to sparse deformable convolution.

[0027] The specific backbone network can be implemented based on the commonly used backbone network models in the prior art, such as VGG, Resnet, DeseNet, etc. In this embodiment, a backbone network is constructed based on the CSPDarknet model, and the convolution method in the CSPDarknet model is modified to sparse deformable convolution, which effectively improves the calculation efficiency, reduces the amount of calculation, and ensures the detection accuracy.

[0028] The head module processes the features extracted by the backbone to complete target detection and depth prediction; each convolution mode in the head module is set to sparse deformable convolution; The detected target is recorded as the predicted target, and the output of the head module includes: the position of each predicted target in the input image, the category of the predicted target, and the distance between the predicted target and the camera.

[0029] like Figure 3 As shown, the structure of the head module in the present application includes: three parallel prediction channels; The three prediction channels are: bbox pred, which is the channel for predicting the position of the target in the image; class pred, which is the channel for predicting the target category; and distance pred, which is the channel for predicting the distance between the target and the camera. Each prediction channel includes: a ConvModule and a Conv2d connected in sequence.

[0030] Among them, ConvModule is the basic module for convolution operation in the deep learning framework, which is mainly used to extract and process image features to detect and recognize targets. Specifically, it includes convolution layer (Conv), batch normalization (BN) and activation function. conv2d is a 2D convolution.

[0031] S6: training the distance measurement model based on the training data set to obtain a trained distance measurement model; During training, the head module uses Loss for the loss function of distance prediction dis function; ; Among them, d pre represents the distance predicted by the network, d gt Indicates the true distance, d max It represents the maximum distance in the training data, and a is a hyperparameter obtained through training.

[0032] In the neural network training process, the loss function is used to evaluate the difference or error between the predicted value and the true label. Generally, the smaller the loss function value, the closer the predicted result is to the true value, and the better the model performance. The distance measurement model in this application is mainly used to predict the distance between the target and the camera, and also outputs the location and category of the target. Therefore, this application uses the distance between the target and the camera as an indicator to measure the effect of model training. pre d gt d max Three distance values ​​are used to construct the loss function. pre ≥d max When , the loss function is not calculated. pre and d gt The closer they are, the better the training effect is, which effectively improves the training convergence speed of the model.

[0033] The specific construction details of the ranging model as well as the training method and process are implemented based on existing technologies.

[0034] The model trained based on this method is suitable for vehicle-mounted cameras with different parameters. It can not only obtain the distance of the target but also detect the position of the target relative to the vehicle. At the same time, the lightweight design enables the model to achieve better real-time effects.

[0035] S7: collect real-time images through the on-board monocular fisheye camera on the vehicle to be measured, which is recorded as: image to be detected; After the distortion of the image to be detected is corrected, a corrected image to be detected is obtained.

[0036] S8: The corrected image to be detected is sent to the trained ranging model in real time to obtain the predicted targets in the image to be detected, as well as the position of each predicted target in the input image, the category of the predicted target, and the distance between the predicted target and the camera.

[0037] S9: Obtain the position of each predicted target in the input image and the distance between the predicted target and the camera, and convert the image coordinates to the world coordinate system according to the internal and external parameters of the camera, that is, obtain the orientation of the predicted target relative to the vehicle to be measured. According to the result output by the ranging model in this application, the relative position between the predicted target and the vehicle where the camera is located can be calculated. The internal parameters of the on-board camera are constants, which have been calibrated and stored in the vehicle before the vehicle goes online. The external parameters are the posture of the on-board camera in the world coordinate system, which describes the relative position relationship between the camera coordinates and the world coordinate system. After obtaining the relative position relationship between the camera head coordinates and the world coordinate system, the distance between the predicted target and the camera can be used to obtain the relative orientation relationship between the predicted target and the vehicle where the camera is located. The specific calculation details such as the conversion process of the internal and external parameters can be implemented based on existing technologies such as calibration algorithms and three-dimensional positioning methods.

[0038] After using the technical solution of the present invention, the distances of targets in images taken by cameras with different parameters are unified into the same focal length space, so that the trained model is applicable to vehicle-mounted cameras with different parameters; a lightweight network structure is designed based on sparse deformable convolution, so that the model reasoning speed reaches real-time effect, and the influence of too fast vehicle speed on target ranging is reduced; a network with target detection and ranging functions is designed, which can not only obtain the distance of the target, but also determine the position of the target relative to the vehicle according to the target detection result and the camera parameters.

Claims

1. A monocular distance measurement method based on a vehicle-mounted fisheye camera, characterized in that: It includes the following steps: S1: Collect images based on a monocular fisheye camera to obtain images to be processed; Perform distortion correction on the image to be processed to obtain a corrected image to be processed; S2: annotating the corrected image to be processed, annotating all preset targets included in the image and their actual distances d relative to the monocular fisheye camera, and obtaining the annotated image to be processed; S3: Based on the annotated images to be processed, build training sets and validation sets; S4: Build a lightweight ranging model; The ranging model includes: a backbone network and a head module connected in sequence; The backbone network realizes the extraction of image features of the input image; the head module processes the features extracted by the backbone to complete target detection and depth prediction; The convolution modes in the backbone network and the head module are both set to sparse deformable convolution; The detected target is recorded as a predicted target, and the output of the head module includes: the position of each predicted target in the input image, the category of the predicted target, and the distance between the predicted target and the camera; S6: Training the distance measurement model based on the training data set to obtain the trained distance measurement model; S7: collect real-time images through the on-board monocular fisheye camera on the vehicle to be measured, which is recorded as: image to be detected; After performing distortion correction on the image to be detected, a corrected image to be detected is obtained; S8: Send the corrected image to be detected to the trained distance measurement model in real time to obtain the predicted targets in the image to be detected, as well as the position of each predicted target in the input image, the category of the predicted target, and the distance between the predicted target and the camera.

2. The monocular distance measurement method based on a vehicle-mounted fisheye camera according to claim 1, characterized in that: The expression of the sparse deformable convolution is: y i =wx+b i ; Among them, y i is the i-th feature point of the local region of each convolution output in the current layer, where i < N, N is the total number of feature points output by each convolution calculation of the sparse deformable convolution, and N > 1; x is the output of the previous layer of the network, w is the network weight of the current layer, and b is the bias item corresponding to the i-th feature point.

3. The monocular distance measurement method based on a vehicle-mounted fisheye camera according to claim 1, characterized in that: In step S6, during the training process, the head module uses Loss as the loss function for distance prediction. dis function; ； Among them, d pre Represents the distance predicted by the network, d gt Indicates the true distance, d max Represents the maximum distance in the training data, and a is a hyperparameter.

4. The monocular distance measurement method based on a vehicle-mounted fisheye camera according to claim 1, characterized in that: It also includes the following steps: S9: Obtain the position of each predicted target in the input image and the distance between the predicted target and the camera, and convert the image coordinates to the world coordinate system according to the internal and external parameters of the camera, so as to obtain the orientation of the predicted target relative to the vehicle to be measured.

5. The monocular distance measurement method based on a vehicle-mounted fisheye camera according to claim 1, characterized in that: It also includes a distance measurement method for realizing a vehicle-mounted camera compatible with different parameters, which specifically includes the following steps: In step S1, images are collected respectively based on a monocular fisheye camera with a plurality of different parameters to obtain images to be processed; Before the implementation of step S3, the following steps are required: According to the focal length f, the images captured by different cameras are mapped to the same focal length space, and the marked d is corrected using the following calculation method: d * = (d*f) / 1000; Among them, d * is the true value of the target distance in the unified focal length space; Use d * The value of is used to mark the actual distances between all preset targets in the image and their corresponding monocular fisheye cameras, and the marked image to be processed is obtained.

6. The monocular distance measurement method based on a vehicle-mounted fisheye camera according to claim 1, characterized in that: The backbone network is constructed based on the CSPDarknet model, and the convolution mode in the CSPDarknet model is modified to the sparse deformable convolution.

7. The monocular distance measurement method based on a vehicle-mounted fisheye camera according to claim 1, characterized in that: In the sparse deformable convolution, the number N of local feature points output by each convolution is 4, i∈(0,1,2,3).

8. The monocular distance measurement method based on a vehicle-mounted fisheye camera according to claim 1, characterized in that: The structure of the head module includes: three parallel prediction channels; The three prediction channels are: a channel for predicting the position of a target in an image, a channel for predicting the target category, and a channel for predicting the distance between a target and a camera; Each of the prediction channels includes: a ConvModule and a Conv2d connected in sequence.

9. The monocular distance measurement method based on a vehicle-mounted fisheye camera according to claim 1, characterized in that: In step S1, the distortion of the image to be processed is compensated based on the homography matrix and the affine transformation.

Images