A method and system for adaptive viewing angle of multiple rear-mounted cameras

Abstract

The present invention relates to the field of driving perception technology, and specifically discloses a method and system for adaptive viewing angles of rear-mounted cameras of multiple vehicles, the method comprising calibrating the on-board camera; acquiring the RGB image collected by the on-board camera in real time, calculating the vanishing point position in the RGB image based on a neural network, and obtaining the actual vanishing point position; calculating the actual elevation angle of the on-board camera according to the actual vanishing point position and the reference vanishing point position, and updating the rotation matrix according to the actual elevation angle; and constructing the conversion relationship between the road surface coordinate point and the corresponding coordinate point on the image according to the parameter matrix, the updated rotation matrix and the translation vector. The present invention utilizes a deep neural network to overcome the influence of weather, environment, illumination, and imaging quality on the recognition of vanishing points in images, and can accurately recognize vanishing points even in conditions without lane lines. At the same time, the mean is used to smooth the vanishing point position, reduce the interference of abnormal recognition, and improve the robustness of vanishing point recognition.

Description

Technical Field

[0001] The present invention relates to the field of driving perception technology, and in particular to a method and system for self-adapting viewing angles of multiple rear-mounted cameras on vehicles. Background Art

[0002] At present, most of the on-board cameras used to identify road diseases and road spills are aftermarket devices, and the installation positions on different vehicles are inconsistent. When it is necessary to perform inverse perspective changes on images collected by multiple on-board cameras, it is necessary to record and calibrate the installation position of each camera, and establish and record the rotation perspective matrix separately, which will generate a lot of calibration and storage work. In addition, when the vehicle is driving, especially when driving at high speeds, the shaking of the vehicle may cause temporary or long-term changes in the perspective of the on-board camera (such as changes in the pitch angle of the camera), destroying the original perspective transformation relationship and causing errors in the target size calculation and distance calculation. Summary of the invention

[0003] The object of the present invention is to provide a method and system for adaptive viewing angles of multiple rear-mounted cameras on vehicles to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A method for adaptive viewing angles of rear-mounted cameras of multiple vehicles, the method comprising:

[0006] Calibrate the vehicle-mounted camera to obtain the rotation matrix, translation vector, pitch angle and reference vanishing point position of the vehicle-mounted camera in the standard state;

[0007] Acquire an RGB image captured by a vehicle-mounted camera in real time, calculate the vanishing point position in the RGB image based on a neural network, and obtain the actual vanishing point position;

[0008] Calculate the actual elevation angle of the onboard camera according to the actual vanishing point position and the reference vanishing point position, and update the rotation matrix according to the actual elevation angle;

[0009] According to the parameter matrix, the updated rotation matrix and translation vector construct the transformation relationship between the road coordinate point and the corresponding coordinate point on the image;

[0010] When the position of the target point on the image is input, the coordinate position of the target point on the road plane is calculated based on the conversion relationship.

[0011] As a further solution of the present invention: the step of calibrating the vehicle-mounted camera to obtain the rotation matrix, translation vector, pitch angle and reference vanishing point position of the vehicle-mounted camera in a standard state includes:

[0012] Performing internal parameter calibration on the vehicle-mounted camera, calculating the internal parameter matrix of the vehicle-mounted camera, and storing the internal parameter matrix according to the model of the vehicle-mounted camera; wherein vehicle-mounted cameras of the same model share the same internal parameter matrix;

[0013] Record the world coordinates of the N square corner points of the checkerboard calibration object and their imaging coordinates in the image, and establish the projection relationship between the N feature points on the checkerboard in the world coordinate system and the N image points in the image based on the preset PnP algorithm;

[0014] Get the rotation matrix, translation vector and pitch angle of the vehicle-mounted camera in the standard state; among them, the rotation matrix, translation vector and pitch angle of the vehicle-mounted cameras with the same height and model are the same;

[0015] The vanishing point position of the image in the calibration state is calculated based on the trained vanishing point recognition neural network to obtain the reference vanishing point position.

[0016] As a further solution of the present invention: the step of acquiring the RGB image captured by the vehicle-mounted camera in real time, calculating the vanishing point position in the RGB image based on a neural network, and obtaining the actual vanishing point position includes:

[0017] Acquire the RGB image captured by the vehicle-mounted camera in real time, and calculate the vanishing point position in the RGB image based on the trained vanishing point recognition neural network;

[0018] Store the vanishing point position in the RGB image into a preset vanishing point queue;

[0019] Read the vanishing point positions of n images in the vanishing point queue, calculate the position mean, and get the actual vanishing point position.

[0020] As a further solution of the present invention: the step of calculating the actual elevation angle of the vehicle-mounted camera according to the actual vanishing point position and the reference vanishing point position, and updating the rotation matrix according to the actual elevation angle comprises:

[0021] Compare the actual vanishing point position with the reference vanishing point position when calibrated under the corresponding camera model and installation height;

[0022] The average pitch angle of the current camera is calculated based on a preset empirical formula to obtain the actual elevation angle;

[0023] An increment of the actual elevation angle relative to the pitch angle is calculated, and a current rotation matrix is ​​obtained according to the increment.

[0024] As a further solution of the present invention: the step of constructing the conversion relationship between the road surface coordinate point and the corresponding coordinate point on the image according to the parameter matrix, the updated rotation matrix and the translation vector comprises:

[0025] Read the parameter matrix of the vehicle camera, the current rotation matrix and the translation vector, and determine the coordinate transformation formula based on the parameter matrix, the current rotation matrix and the translation vector;

[0026] Input the world coordinates of N points in the calibration process into the coordinate conversion formula to obtain their coordinate positions on the corresponding image;

[0027] The homography matrix between the image plane currently collected by the vehicle-mounted camera and the road surface is calculated based on the coordinate position and the world coordinates, and the conversion relationship between the road surface coordinate point and the corresponding coordinate point on the image is constructed based on the homography matrix.

[0028] As a further solution of the present invention: the method further comprises:

[0029] Input the acquired RGB image into the trained anomaly analysis model to obtain the abnormal area;

[0030] The image size of the abnormal area is obtained, and the image size is input into the conversion relationship to obtain the actual size.

[0031] The technical solution of the present invention also provides a multi-vehicle rear-mounted camera viewing angle adaptive system, the system comprising:

[0032] The standard parameter acquisition module is used to calibrate the vehicle-mounted camera and obtain the rotation matrix, translation vector, pitch angle and reference vanishing point position of the vehicle-mounted camera in the standard state;

[0033] An image acquisition module is used to acquire an RGB image acquired by a vehicle-mounted camera in real time, calculate the vanishing point position in the RGB image based on a neural network, and obtain the actual vanishing point position;

[0034] A parameter updating module, used to calculate the actual elevation angle of the vehicle-mounted camera according to the actual vanishing point position and the reference vanishing point position, and update the rotation matrix according to the actual elevation angle;

[0035] A transformation relationship generation module, used to construct a transformation relationship between a road surface coordinate point and a corresponding coordinate point on an image according to a parameter matrix, an updated rotation matrix and a translation vector;

[0036] The transformation relationship application module is used to calculate the coordinate position of the target point on the road plane based on the transformation relationship when the position of the target point on the input image is obtained.

[0037] As a further solution of the present invention: the standard parameter acquisition module includes:

[0038] An internal parameter calibration unit, used to perform internal parameter calibration on the vehicle-mounted camera, calculate the internal parameter matrix of the vehicle-mounted camera, and store the internal parameter matrix according to the model of the vehicle-mounted camera; wherein vehicle-mounted cameras of the same model share the same internal parameter matrix;

[0039] A projection relationship establishing unit is used to record the world coordinates of the N square corner points of the chessboard calibration object and their imaging coordinates in the image, and to establish the projection relationship between the N feature points on the chessboard in the world coordinate system and the N image points in the image based on a preset PnP algorithm;

[0040] A data acquisition unit, used to acquire the rotation matrix, translation vector and pitch angle of the vehicle-mounted camera in a standard state; wherein the rotation matrix, translation vector and pitch angle of the vehicle-mounted cameras with the same height and model are the same;

[0041] The reference vanishing point calculation unit is used to calculate the vanishing point position of the image in the calibration state based on the trained vanishing point recognition neural network to obtain the reference vanishing point position.

[0042] As a further solution of the present invention: the image acquisition module includes:

[0043] An actual vanishing point calculation unit, used to obtain the RGB image captured by the vehicle-mounted camera in real time, and calculate the vanishing point position in the RGB image based on the trained vanishing point recognition neural network;

[0044] A vanishing point pre-storage unit, used for storing the vanishing point position in the RGB image into a preset vanishing point queue;

[0045] The mean calculation unit is used to read the vanishing point positions of n images in the vanishing point queue, calculate the position mean, and obtain the actual vanishing point position.

[0046] As a further solution of the present invention: the parameter updating module includes:

[0047] A comparison unit, used for comparing the actual vanishing point position with the reference vanishing point position when calibrated under the corresponding camera model and installation height;

[0048] The actual elevation angle calculation unit is used to calculate the average pitch angle of the current camera based on a preset empirical formula to obtain the actual elevation angle;

[0049] The increment calculation and application unit is used to calculate the increment of the actual elevation angle relative to the pitch angle, and obtain the current rotation matrix according to the increment calculation.

[0050] Compared with the prior art, the present invention has the following beneficial effects: the present invention utilizes a deep neural network to overcome the influence of weather, environment, lighting, and imaging quality on the recognition of vanishing points in images, and can accurately identify vanishing points even in conditions without lane lines. At the same time, the mean is used to smooth the position of the vanishing points, thereby reducing the interference of abnormal recognition and improving the robustness of vanishing point recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0052] Figure 1 The flowchart of the multi-vehicle rear-mounted camera view adaptation method.

[0053] Figure 2 This is the first sub-process flowchart of the method for adaptively adjusting the viewing angle of rear-mounted cameras of multiple vehicles.

[0054] Figure 3 This is a second sub-process flowchart of the method for adaptively adjusting the viewing angles of rear-mounted cameras of multiple vehicles.

[0055] Figure 4 This is the third sub-process flowchart of the multi-vehicle rear-mounted camera perspective adaptation method.

[0056] Figure 5 This is the fourth sub-process flowchart of the method for adaptively adjusting the viewing angles of rear-mounted cameras of multiple vehicles.

[0057] Figure 6 This is a structural block diagram of the multi-vehicle rear-mounted camera perspective adaptive system. DETAILED DESCRIPTION

[0058] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] Example 1

[0060] Figure 1 : is a flowchart of a method for self-adapting the viewing angles of rear-mounted cameras of multiple vehicles. In an embodiment of the present invention, a method for self-adapting the viewing angles of rear-mounted cameras of multiple vehicles comprises steps S100 to S500:

[0061] Step S100: calibrate the vehicle-mounted camera to obtain the rotation matrix, translation vector, pitch angle and reference vanishing point position of the vehicle-mounted camera in a standard state;

[0062] Step S200: acquiring an RGB image captured by a vehicle-mounted camera in real time, calculating the vanishing point position in the RGB image based on a neural network, and obtaining an actual vanishing point position;

[0063] Step S300: calculating the actual elevation angle of the vehicle-mounted camera according to the actual vanishing point position and the reference vanishing point position, and updating the rotation matrix according to the actual elevation angle;

[0064] Step S400: constructing a transformation relationship between the road surface coordinate point and the corresponding coordinate point on the image according to the parameter matrix, the updated rotation matrix and the translation vector;

[0065] Step S500: when the position of the target point on the image is input, the coordinate position of the target point on the road plane is calculated based on the conversion relationship;

[0066] The camera device on the vehicle can be integrated or split. The integrated device is a camera and a central industrial computer integrated into one, similar to a driving recorder; the split device includes a separately installed vehicle-mounted camera device and a central industrial computer.

[0067] The vehicle-mounted camera device is a vehicle-mounted high-definition single-channel camera, which is powered by a central industrial control computer or a vehicle-mounted camera. The camera has a pixel of more than one million pixels and a clear image quality. It is used to capture images in front of or behind the vehicle. It can be installed inside or outside the vehicle. The image acquisition frequency can be adjusted based on the vehicle speed or based on the acquisition signal sent by the central industrial control computer. Specifically, the adjustment based on the vehicle speed is that the acquisition system obtains the real-time speed of the vehicle through the positioning module, and dynamically adjusts the image acquisition frequency according to the speed value; the adjustment based on the central industrial control computer is to use the central industrial control computer to connect to the rotary encoder installed on the wheel. The encoder sends an instruction to capture the image to the vehicle-mounted camera device based on the determined distance value of the vehicle.

[0068] The central industrial computer is installed inside the vehicle and provides power for the vehicle. It has the fourth generation (4G) or fifth generation (5G) mobile communication technology, a high-precision positioning system and an information collection system. It can send instructions to the on-board camera device to collect images, and continuously receive, process and upload the collected data. In addition, the central industrial computer can also contain a micro-graphics processing unit (GPU), and the deep neural network algorithm can use the GPU to accelerate image processing.

[0069] Install a camera on the vehicle to collect images around the vehicle. Using intelligent algorithms, the target objects in the image can be identified, including road damage, road spills, pedestrians, vehicles, etc. Since the collected images have obvious perspective, that is, the pixels at different positions in the image represent different physical sizes, in order to further describe the properties of the identified target (such as the area of ​​the disease, the size of the spills, the distance of pedestrians and vehicles), it is generally necessary to reverse the perspective of the collected image and convert it into a bird's-eye view perpendicular to the road surface, and normalize the pixel units on the bird's-eye view to establish a connection between the pixels and the real physical size. According to the number of pixels and pixel distance occupied by the identified target in the bird's-eye view, the real size and real distance of the target can be calculated.

[0070] In order to realize the reverse perspective of the image, it is necessary to establish the relationship between the world coordinate system, camera coordinate system, image coordinate system, pixel coordinate system, etc., so that the image points can be perspectively projected into the world coordinate system through transformations such as rotation, translation, affine, and perspective. This calculation requires knowing the position and angle of the camera in the world coordinate system. In addition, the camera needs to be calibrated internally to obtain parameters such as the focal length and distortion coefficient of the camera lens.

[0071] Figure 2 This is a first sub-flow chart of the method for adaptive viewing angle of rear-mounted cameras on multiple vehicles. The step of calibrating the vehicle-mounted cameras to obtain the rotation matrix, translation vector, pitch angle and reference vanishing point position of the vehicle-mounted cameras in a standard state includes steps S101 to S104:

[0072] Step S101: calibrating the internal parameters of the vehicle-mounted camera, calculating the internal parameter matrix of the vehicle-mounted camera, and storing the internal parameter matrix according to the model of the vehicle-mounted camera; wherein vehicle-mounted cameras of the same model share the same internal parameter matrix;

[0073] Step S102: Record the world coordinates of the N square corner points of the checkerboard calibration object and their imaging coordinates in the image, and establish the projection relationship between the N feature points on the checkerboard in the world coordinate system and the N image points in the image based on a preset PnP algorithm;

[0074] Step S103: Obtain the rotation matrix, translation vector and pitch angle of the vehicle-mounted camera in a standard state; wherein the rotation matrix, translation vector and pitch angle of the vehicle-mounted cameras with the same height and model are the same;

[0075] Step S104: Calculate the vanishing point position of the image in the calibration state based on the trained vanishing point recognition neural network to obtain a reference vanishing point position.

[0076] Steps S101 to S104 specifically define the calibration process of the vehicle-mounted camera. After the vehicle-mounted camera is installed, a checkerboard calibration object is placed on a flat road surface within the camera's shooting range. The checkerboard calibration object is placed horizontally on the road surface and consists of black and white squares. The effective length and width of the checkerboard calibration object are not less than 1 meter. The checkerboard calibration object can be clearly seen in the image collected by the vehicle-mounted camera, and the square corners in the checkerboard can be clearly seen. The area occupied by the checkerboard calibration object in the image is not less than 15,000 pixels. Record the world coordinates [Xw, Yw, Zw] of the N square corners of the chessboard calibration object and their imaging coordinates [u, v] in the image, use the PnP (Perspective-n-Point) algorithm to establish the projection relationship between the N square grid points on the chessboard in the world coordinate system and the N image points in the image imaging, obtain the rotation matrix rvecbenc, translation vector tvecbenc, and the current corresponding pitch angle αbenc of the camera in the standard state, and use a deep neural network to identify the vanishing point position (ubenc, vbenc) in the image, ubenc is the horizontal pixel position of the vanishing point, and vbenc is the vertical pixel position of the vanishing point. Since the chessboard is placed horizontally on the road surface, the world coordinates of the square corners can generally be set to 0 in the z-axis direction. The N square grid points generally refer to N points that are not collinear in the chessboard calibration object, N ≥ 4. After calibration, record the height of the camera from the ground. Cameras of the same model and installed at the same height share the same rotation matrix, translation vector, pitch angle, and standard image vanishing point position.

[0077] Figure 3 The second sub-process flowchart of the method for adaptive viewing angle of rear-mounted cameras of multiple vehicles is shown in FIG. 1 , wherein the step of acquiring the RGB image captured by the vehicle-mounted camera in real time, calculating the vanishing point position in the RGB image based on a neural network, and obtaining the actual vanishing point position includes steps S201 to S203:

[0078] Step S201: acquiring an RGB image captured by a vehicle-mounted camera in real time, and calculating the vanishing point position in the RGB image based on a trained vanishing point recognition neural network;

[0079] Step S202: storing the vanishing point position in the RGB image into a preset vanishing point queue;

[0080] Step S203: Read the vanishing point positions of n images in the vanishing point queue, calculate the position mean, and obtain the actual vanishing point position.

[0081] Steps S201 to S203 specifically define the process of determining the actual vanishing point position:

[0082] 1) After calibration, collect images while the vehicle is driving.

[0083] 2) Use a deep neural network to calculate the vanishing point position (ui, vi) in the image, where ui is the horizontal pixel position of the vanishing point and vi is the vertical pixel position of the vanishing point, and store (ui, vi) in the vanishing point queue VP_list.

[0084] 3) Take the vanishing point positions of the nearly n images in the vanishing point queue VP_list and calculate the average to get the average vanishing point position (uave, vave), where uave is the horizontal pixel position of the average vanishing point, and vave is the vertical pixel position of the average vanishing point.

[0085] Figure 4 The third sub-flow chart of the method for adaptive viewing angle of rear-mounted cameras of multiple vehicles, wherein the step of calculating the actual elevation angle of the vehicle-mounted camera according to the actual vanishing point position and the reference vanishing point position, and updating the rotation matrix according to the actual elevation angle includes steps S301 to S303:

[0086] Step S301: comparing the actual vanishing point position with the reference vanishing point position when calibrated at the corresponding camera model and installation height;

[0087] Step S302: Calculate the average pitch angle of the current camera based on a preset empirical formula to obtain the actual elevation angle;

[0088] Step S303: Calculate the increment of the actual elevation angle relative to the pitch angle, and obtain the current rotation matrix according to the increment.

[0089] Steps S301 to S303 specifically define the updating process of the rotation matrix:

[0090] 4) Compare the average vanishing point position (uave, vave) with the calibrated vanishing point position (ubenc, vbenc) of the corresponding camera model and installation height;

[0091] 5) Use the empirical formula vbenc-vave=β(αbenc-αave) , calculate the average pitch angle αave of the current camera. Among them, β is the proportional coefficient of the change in the vertical height position of the vanishing point and the change in the camera pitch angle, which is obtained through a large number of comparative test statistics, verifying that under the same camera and the same installation height, the vertical height position change of the image vanishing point is positively correlated with the change in the camera pitch angle.

[0092] 6) Use the camera's new average pitch angle αave Pitch angle relative to the calibration state The incremental part of αbenc is used to calculate the current rotation matrix rvecnew of the camera.

[0093] Figure 5This is a fourth sub-flow chart of the method for adaptive viewing angle of rear-mounted cameras of multiple vehicles. The step of constructing the conversion relationship between the road coordinate point and the corresponding coordinate point on the image according to the parameter matrix, the updated rotation matrix and the translation vector includes steps S401 to S403:

[0094] Step S401: Read the parameter matrix, current rotation matrix and translation vector of the vehicle-mounted camera, and determine the coordinate transformation formula according to the parameter matrix, current rotation matrix and translation vector;

[0095] Step S402: inputting the world coordinates of N points in the calibration process into the coordinate conversion formula to obtain their coordinate positions on the corresponding image;

[0096] Step S403: Calculate the homography matrix between the image plane currently captured by the vehicle-mounted camera and the road surface based on the coordinate position and the world coordinates, and construct a conversion relationship between the road surface coordinate point and the corresponding coordinate point on the image based on the homography matrix.

[0097] 7) Use the intrinsic parameter matrix K of the corresponding camera model, the current rotation matrix rvecnew of the camera, and the translation vector tvecbenc, according to the formula:

[0098]

[0099] Input the world coordinates [Xw, Yw, Zw] of the N points in the calibration process, and obtain their coordinate positions [up, vp] on the new corresponding image. Zc is the position of the point on the Z axis of the camera coordinate system, which can be obtained by the Euclidean distance between the camera and the point.

[0100] 8) [up, vp] and [Xw, Yw, Zw] can be used to calculate the homography matrix Hnew between the current camera-captured image plane and the road surface. According to the formula [xw, yw, zw]=Hnew[U, V], the transformation relationship between the road surface coordinate point and the corresponding coordinate point on the image is constructed, that is, the position of the target point on the input image [U, V] is used to calculate the coordinate position of the target point on the road plane [xw, yw, zw], and then the real size and real distance of the target object can be calculated.

[0101] Furthermore, the method further comprises:

[0102] Input the acquired RGB image into the trained anomaly analysis model to obtain the abnormal area;

[0103] The image size of the abnormal area is obtained, and the image size is input into the conversion relationship to obtain the actual size.

[0104] 9) The algorithm and system can be used in conjunction with road disease identification, road foreign object identification, vehicle and pedestrian identification, etc. to help accurately calculate the true size and distance of the target object (such as the length of road cracks, the size of road foreign objects, the distance between the target vehicle and the target pedestrian and the inspection vehicle). The algorithm and system can be deployed on the edge or in the cloud.

[0105] The technical solution of the present invention provides an adaptive algorithm and system for the perspective of multiple rear-mounted cameras on vehicles based on a deep neural network. The algorithm uses a deep neural network to identify vanishing points, and uses the positive correlation between the position change of the image vanishing point and the position change of the camera pitch angle to dynamically update the image inverse perspective transformation relationship in real time during the acquisition process, reduce vehicle jitter during movement or human interference with the camera perspective transformation, and help accurately calculate and identify the size and distance of the target object.

[0106] Using this algorithm and system, cameras of the same model can share the same camera intrinsic parameter matrix K; under the conditions of the same camera model and the same installation height, the same camera extrinsic parameters (including rotation matrix rvecbenc, translation vector tvecbenc, and the current corresponding pitch angle αbenc, vanishing point position (ubenc, vbenc)) can be shared, which can greatly reduce the camera calibration work when installed on multiple vehicles and realize rapid and large-scale installation of equipment.

[0107] Compared with the method of identifying image vanishing points based on traditional machine vision, the use of deep neural networks can overcome the influence of weather, environment, lighting, and imaging quality on the identification of vanishing points in images. It can even accurately identify vanishing points in conditions without lane lines. At the same time, the mean is used to smooth the position of vanishing points, reduce the interference of abnormal identification, and improve the robustness of vanishing point identification.

[0108] Figure 6 The structural block diagram of the multi-vehicle rear-mounted camera perspective adaptive system is shown. In an example of the technical solution of the present invention, the vehicle-mounted camera device is connected to the central industrial control computer, and the central industrial control computer sends an image acquisition instruction to the vehicle-mounted camera device. The central industrial control computer is powered by the detection vehicle. The specific rear-mounted camera adaptive algorithm is deployed on the central industrial control computer or deployed in the cloud, specifically:

[0109] (1) Perform intrinsic calibration on the vehicle-mounted camera used and calculate the camera’s intrinsic parameter matrix K. Cameras of the same model share the same intrinsic parameter matrix and do not need to be calibrated repeatedly.

[0110] (2) Record the installation height of the camera. After the camera is installed, let the camera collect a picture of the chessboard on the road ahead. Use the PnP algorithm to establish the projection relationship between the N feature points [Xw, Yw, Zw] on the chessboard in the world coordinate system and the N image points [u, v] in the image imaging, and obtain the rotation matrix rvecbenc, translation vector tvecbenc, and current corresponding pitch angle of the camera in the standard state. αbenc, using neural network to calculate the vanishing point position of the image under calibration state (ubenc, vbenc)

[0111] (3) When multiple rear-mounted cameras use the same camera internal and external parameters, the camera models must be the same and the installation height must be controlled within a certain error range.

[0112] (4) The rear camera of the patrol vehicle collects RGB images, uses the vanishing point recognition deep neural network to calculate the vanishing point position (ui, vi) in the image, and stores it in the vanishing point queue VP_list. The vanishing point positions of the last n images in the vanishing point queue VP_list are taken to calculate the average, and the average image vanishing point position (uave, vave) is obtained.

[0113] (5) Compare the average vanishing point position (uave, vave) with the reference vanishing point position (ubenc, vbenc): Use the empirical formula vbenc-vave=β(αbenc-αave) to calculate the average pitch angle αave of the camera when nearly n images are collected. β is the proportional coefficient of the change in the longitudinal height position of the vanishing point to the change in the pitch angle of the camera, which can be obtained through a large number of comparative test statistics.

[0114] (6) Use the camera internal parameter matrix K of the corresponding model and installation height, the new rotation matrix rvecnew of the camera relative to the N feature points on the chessboard, and the translation vector tvecbenc, according to the formula:

[0115]

[0116] The image coordinates [up, vp] corresponding to the N feature points [Xw, Yw, Zw] in the new image can be calculated.

[0117] (7) Using [up, vp] and [Xw, Yw, Zw], the homography matrix Hnew between the current camera-captured image plane and the road surface can be calculated. According to the formula [xw, yw, zw] = Hnew[U, V], the transformation relationship between the road surface coordinate point and the corresponding coordinate point on the image is constructed, that is, the position of the target point on the input image [U, V] is used to calculate the coordinate position of the target point on the road plane [xw, yw, zw].

[0118] (8) Based on (7), the recognition of road surface defects, foreign objects, vehicles, and pedestrians can be used to help accurately calculate the true size and distance of the target object (such as the length of road cracks, the size of foreign objects, and the distance between the target vehicle and the inspection vehicle).

[0119] Example 2

[0120] Different from the first embodiment, in the embodiment of the present invention, a multi-vehicle rear-mounted camera viewing angle adaptive system is provided, and the system includes:

[0121] The standard parameter acquisition module is used to calibrate the vehicle-mounted camera and obtain the rotation matrix, translation vector, pitch angle and reference vanishing point position of the vehicle-mounted camera in the standard state;

[0122] An image acquisition module is used to acquire an RGB image acquired by a vehicle-mounted camera in real time, calculate the vanishing point position in the RGB image based on a neural network, and obtain the actual vanishing point position;

[0123] A parameter updating module, used to calculate the actual elevation angle of the vehicle-mounted camera according to the actual vanishing point position and the reference vanishing point position, and update the rotation matrix according to the actual elevation angle;

[0124] A transformation relationship generation module, used to construct a transformation relationship between a road surface coordinate point and a corresponding coordinate point on an image according to a parameter matrix, an updated rotation matrix and a translation vector;

[0125] The transformation relationship application module is used to calculate the coordinate position of the target point on the road plane based on the transformation relationship when the position of the target point on the input image is obtained.

[0126] 8. The multi-vehicle rear-mounted camera viewing angle adaptive system according to claim 7, characterized in that the standard parameter acquisition module comprises:

[0127] An internal parameter calibration unit, used to perform internal parameter calibration on the vehicle-mounted camera, calculate the internal parameter matrix of the vehicle-mounted camera, and store the internal parameter matrix according to the model of the vehicle-mounted camera; wherein vehicle-mounted cameras of the same model share the same internal parameter matrix;

[0128] A projection relationship establishing unit is used to record the world coordinates of the N square corner points of the chessboard calibration object and their imaging coordinates in the image, and to establish the projection relationship between the N feature points on the chessboard in the world coordinate system and the N image points in the image based on a preset PnP algorithm;

[0129] A data acquisition unit, used to acquire the rotation matrix, translation vector and pitch angle of the vehicle-mounted camera in a standard state; wherein the rotation matrix, translation vector and pitch angle of the vehicle-mounted cameras with the same height and model are the same;

[0130] The reference vanishing point calculation unit is used to calculate the vanishing point position of the image in the calibration state based on the trained vanishing point recognition neural network to obtain the reference vanishing point position.

[0131] 9. The multi-vehicle rear-mounted camera viewing angle adaptive system according to any one of claim 7, characterized in that the image acquisition module comprises:

[0132] An actual vanishing point calculation unit, used to obtain the RGB image captured by the vehicle-mounted camera in real time, and calculate the vanishing point position in the RGB image based on the trained vanishing point recognition neural network;

[0133] A vanishing point pre-storage unit, used for storing the vanishing point position in the RGB image into a preset vanishing point queue;

[0134] The mean calculation unit is used to read the vanishing point positions of n images in the vanishing point queue, calculate the position mean, and obtain the actual vanishing point position.

[0135] 10. The multi-vehicle rear-mounted camera perspective adaptive system according to any one of claim 7, wherein the parameter updating module comprises:

[0136] A comparison unit, used for comparing the actual vanishing point position with the reference vanishing point position when calibrated under the corresponding camera model and installation height;

[0137] The actual elevation angle calculation unit is used to calculate the average pitch angle of the current camera based on a preset empirical formula to obtain the actual elevation angle;

[0138] The increment calculation and application unit is used to calculate the increment of the actual elevation angle relative to the pitch angle, and obtain the current rotation matrix according to the increment calculation.

[0139] The functions that can be achieved by the method for adaptively adjusting the viewing angles of rear-mounted cameras of multiple vehicles are all completed by a computer device, and the computer device includes one or more processors and one or more memories, and at least one program code is stored in the one or more memories. The program code is loaded and executed by the one or more processors to implement the method for adaptively adjusting the viewing angles of rear-mounted cameras of multiple vehicles.

[0140] The processor takes out instructions from the memory one by one, analyzes the instructions, and then completes the corresponding operations according to the instruction requirements, generating a series of control commands, so that the various parts of the computer can automatically, continuously and coordinately move to become an organic whole, realize the input of programs, the input of data, and the calculation and output of results. The arithmetic operations or logical operations generated in this process are all completed by the operator; the memory includes a read-only memory (ROM), which is used to store computer programs, and a protection device is provided outside the memory.

[0141] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0142] Those skilled in the art will understand that the description of the above service equipment is merely an example and does not constitute a limitation on the terminal equipment. It may include more or fewer components than described above, or a combination of certain components, or different components, for example, it may include input and output devices, network access devices, buses, etc.

[0143] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire user terminal.

[0144] The above-mentioned memory can be used to store computer programs and / or modules. The above-mentioned processor realizes various functions of the above-mentioned terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an information collection template display function, a product information release function, etc.), etc.; the data storage area can store data created according to the use of the berth status display system (such as product information collection templates corresponding to different product types, product information that different product providers need to release, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0145] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable medium. Based on this understanding, the present invention implements all or part of the modules / units in the above-mentioned embodiment system, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable medium, and the computer program can realize the functions of the above-mentioned various system embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0146] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0147] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for adaptive viewing angle of multiple rear-mounted cameras, characterized in that: The method comprises: Calibrate the vehicle-mounted camera to obtain the rotation matrix, translation vector, pitch angle and reference vanishing point position of the vehicle-mounted camera in the standard state; Acquire an RGB image captured by a vehicle-mounted camera in real time, calculate the vanishing point position in the RGB image based on a neural network, and obtain the actual vanishing point position; Calculate the actual elevation angle of the onboard camera according to the actual vanishing point position and the reference vanishing point position, and update the rotation matrix according to the actual elevation angle; According to the parameter matrix, the updated rotation matrix and translation vector construct the transformation relationship between the road coordinate point and the corresponding coordinate point on the image; When the position of the target point on the input image is calculated, the coordinate position of the target point on the road plane is obtained based on the conversion relationship; The step of calibrating the vehicle-mounted camera to obtain the rotation matrix, translation vector, pitch angle and reference vanishing point position of the vehicle-mounted camera in a standard state includes: Performing internal parameter calibration on the vehicle-mounted camera, calculating the internal parameter matrix of the vehicle-mounted camera, and storing the internal parameter matrix according to the model of the vehicle-mounted camera; wherein vehicle-mounted cameras of the same model share the same internal parameter matrix; Record the world coordinates of the N square corner points of the checkerboard calibration object and their imaging coordinates in the image, and establish the projection relationship between the N feature points on the checkerboard in the world coordinate system and the N image points in the image based on the preset PnP algorithm; Get the rotation matrix, translation vector and pitch angle of the vehicle-mounted camera in the standard state; among them, the rotation matrix, translation vector and pitch angle of the vehicle-mounted cameras with the same height and model are the same; The vanishing point position of the image in the calibration state is calculated based on the trained vanishing point recognition neural network to obtain the reference vanishing point position.

2. The method for adaptive viewing angle of multiple rear-mounted cameras of claim 1, characterized in that: The step of acquiring an RGB image captured by a vehicle-mounted camera in real time, calculating the vanishing point position in the RGB image based on a neural network, and obtaining the actual vanishing point position comprises: Acquire the RGB image captured by the vehicle-mounted camera in real time, and calculate the vanishing point position in the RGB image based on the trained vanishing point recognition neural network; Store the vanishing point position in the RGB image into a preset vanishing point queue; Read the vanishing point positions of n images in the vanishing point queue, calculate the position mean, and get the actual vanishing point position.

3. The method for adaptive viewing angle of multiple rear-mounted cameras of claim 1, characterized in that: The step of calculating the actual elevation angle of the vehicle-mounted camera according to the actual vanishing point position and the reference vanishing point position, and updating the rotation matrix according to the actual elevation angle comprises: Compare the actual vanishing point position with the reference vanishing point position when calibrated under the corresponding camera model and installation height; The average pitch angle of the current camera is calculated based on a preset empirical formula to obtain the actual elevation angle; An increment of the actual elevation angle relative to the pitch angle is calculated, and a current rotation matrix is ​​obtained according to the increment.

4. The method for adaptive viewing angle of multiple rear-mounted cameras according to claim 1, characterized in that: The step of constructing the conversion relationship between the road surface coordinate point and the corresponding coordinate point on the image according to the parameter matrix, the updated rotation matrix and the translation vector comprises: Read the parameter matrix of the vehicle camera, the current rotation matrix and the translation vector, and determine the coordinate transformation formula based on the parameter matrix, the current rotation matrix and the translation vector; Input the world coordinates of N points in the calibration process into the coordinate conversion formula to obtain their coordinate positions on the corresponding image; The homography matrix between the image plane currently collected by the vehicle camera and the road surface is calculated based on the coordinate position and the world coordinates, and the conversion relationship between the road surface coordinate point and the corresponding coordinate point on the image is constructed based on the homography matrix.

5. The method for adaptive viewing angle of multiple rear-mounted cameras of claim 1, characterized in that: The method further comprises: Input the acquired RGB image into the trained anomaly analysis model to obtain the abnormal area; The image size of the abnormal area is obtained, and the image size is input into the conversion relationship to obtain the actual size.

6. A multi-vehicle rear-mounted camera viewing angle adaptive system, used to implement any of the methods described in claims 1-5, characterized in that: The system comprises: The standard parameter acquisition module is used to calibrate the vehicle-mounted camera and obtain the rotation matrix, translation vector, pitch angle and reference vanishing point position of the vehicle-mounted camera in the standard state; An image acquisition module is used to acquire an RGB image acquired by a vehicle-mounted camera in real time, calculate the vanishing point position in the RGB image based on a neural network, and obtain the actual vanishing point position; A parameter updating module, used to calculate the actual elevation angle of the vehicle-mounted camera according to the actual vanishing point position and the reference vanishing point position, and update the rotation matrix according to the actual elevation angle; A transformation relationship generation module, used to construct a transformation relationship between a road surface coordinate point and a corresponding coordinate point on an image according to a parameter matrix, an updated rotation matrix and a translation vector; The transformation relationship application module is used to calculate the coordinate position of the target point on the road plane based on the transformation relationship when the position of the target point on the input image is obtained.

7. The multi-vehicle rear-mounted camera viewing angle adaptive system according to claim 6, characterized in that: The standard parameter acquisition module includes: An internal parameter calibration unit, used to perform internal parameter calibration on the vehicle-mounted camera, calculate the internal parameter matrix of the vehicle-mounted camera, and store the internal parameter matrix according to the model of the vehicle-mounted camera; wherein vehicle-mounted cameras of the same model share the same internal parameter matrix; A projection relationship establishing unit is used to record the world coordinates of the N square corner points of the chessboard calibration object and their imaging coordinates in the image, and to establish the projection relationship between the N feature points on the chessboard in the world coordinate system and the N image points in the image based on a preset PnP algorithm; A data acquisition unit, used to acquire the rotation matrix, translation vector and pitch angle of the vehicle-mounted camera in a standard state; wherein the rotation matrix, translation vector and pitch angle of the vehicle-mounted cameras with the same height and model are the same; The reference vanishing point calculation unit is used to calculate the vanishing point position of the image in the calibration state based on the trained vanishing point recognition neural network to obtain the reference vanishing point position.

8. The multi-vehicle rear-mounted camera viewing angle adaptive system according to claim 6, characterized in that: The image acquisition module comprises: An actual vanishing point calculation unit, used to obtain the RGB image captured by the vehicle-mounted camera in real time, and calculate the vanishing point position in the RGB image based on the trained vanishing point recognition neural network; A vanishing point pre-storage unit, used for storing the vanishing point position in the RGB image into a preset vanishing point queue; The mean calculation unit is used to read the vanishing point positions of n images in the vanishing point queue, calculate the position mean, and obtain the actual vanishing point position.

9. The multi-vehicle rear-mounted camera viewing angle adaptive system according to claim 6, characterized in that: The parameter updating module comprises: A comparison unit, used for comparing the actual vanishing point position with the reference vanishing point position when calibrated under the corresponding camera model and installation height; The actual elevation angle calculation unit is used to calculate the average pitch angle of the current camera based on a preset empirical formula to obtain the actual elevation angle; The increment calculation and application unit is used to calculate the increment of the actual elevation angle relative to the pitch angle, and obtain the current rotation matrix according to the increment calculation.

Images