Intelligent networked bus multi-vehicle platoon perception method and device based on monocular camera

Abstract

The present application relates to the field of autonomous driving technology, and in particular to a method and device for sensing multi-vehicle formations of intelligent networked buses based on a monocular camera, wherein the method includes: detecting the detection frame of the front vehicle in the multi-vehicle formation of the intelligent networked bus on the image, and cutting out the back panel of the front vehicle to detect multiple center points of the front vehicle; calculating the position and yaw angle of the front vehicle in the camera coordinate system, and converting the coordinates of the front vehicle in the camera coordinate system to the vehicle coordinate system of the rear vehicle according to the external parameter matrix obtained by calibrating the camera and the rear vehicle, so as to obtain the coordinates of the front vehicle in the vehicle coordinate system of the rear vehicle, and generate the sensing results of the multi-vehicle formation of the intelligent networked bus. Thus, the problems in the related technology that the overall deployment cost of the vehicle formation technology increases and the accuracy decreases due to the relatively high cost of the laser radar and radar systems and their sensitivity to environmental conditions, and the complexity and maintenance difficulty of the system increase due to the need for precise calibration and synchronization of the multi-camera system.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method and device for sensing a multi-vehicle formation of an intelligent connected bus based on a monocular camera. Background Art

[0002] As urban traffic problems continue to intensify, how to efficiently manage and dispatch public transportation systems has become a pressing issue. Public transportation systems rely on fixed routes and schedules, which makes the system inflexible and unable to respond to changes in traffic demand in a timely manner, especially during peak traffic hours or emergencies. In addition, fixed routes and station settings usually cannot perfectly match the actual needs of passengers, resulting in increased waiting time for passengers and reduced vehicle utilization efficiency.

[0003] Among the related technologies, vehicle platooning technology is mainly used. By controlling a group of vehicles to travel at a fixed speed and interval, the road utilization rate and vehicle driving safety can be effectively improved. It mainly relies on sensors such as lidar, radar, and multi-camera vision systems. It can accurately perceive the position, speed, and relative relationship of each vehicle in the fleet, realize precise vehicle platoon control, and support complex vehicle platooning algorithms.

[0004] However, in the related technologies, due to the relatively high cost of lidar and radar systems and their sensitivity to environmental conditions, the overall deployment cost of vehicle platooning technology may increase and affect its performance and accuracy. For multi-camera based vision systems, the amount of calculation is large when processing complex scenes, the demand for computing resources is high, and the performance degrades significantly when the lighting conditions are poor. In addition, since the multi-camera system requires precise calibration and synchronization, the complexity of the system and the difficulty of maintenance are increased, which urgently needs to be improved. Summary of the invention

[0005] The present application provides a method and device for sensing multi-vehicle platooning of intelligent connected buses based on a monocular camera, in order to solve the problems in the related technology that the overall deployment cost of vehicle platooning technology may increase and affect its performance and accuracy due to the relatively high cost of lidar and radar systems and their sensitivity to environmental conditions, and the multi-camera system requires precise calibration and synchronization, which increases the complexity of the system and the difficulty of maintenance.

[0006] The first aspect of the present application provides a method for perceiving a multi-vehicle formation of an intelligent connected bus based on a monocular camera, comprising the following steps: detecting a detection frame of a front vehicle in a multi-vehicle formation of an intelligent connected bus on an image; based on the detection frame, cropping out a back panel of the front vehicle to detect multiple center points of the front vehicle; calculating a position and a yaw angle of the front vehicle in a camera coordinate system according to the multiple center points, and converting the coordinates of the front vehicle in the camera coordinate system to a vehicle coordinate system of the rear vehicle according to an external parameter matrix obtained by calibrating the camera and the rear vehicle to obtain the coordinates of the front vehicle in the vehicle coordinate system of the rear vehicle, thereby generating a perception result of the multi-vehicle formation of the intelligent connected bus.

[0007] Optionally, in one embodiment of the present application, the back panel of the front vehicle is cropped out based on the detection frame to detect multiple center points of the front vehicle, including: obtaining a picture of the back panel and the position of the back panel in the picture; binarizing the picture to obtain a binarization result; identifying the vehicle contour based on the binarization result, and determining the multiple center points based on the vehicle contour.

[0008] Optionally, in one embodiment of the present application, the position and yaw angle of the leading vehicle in the camera coordinate system are calculated based on the multiple center points, including: calculating the pixel distance from the coordinates of each center point to the horizontal straight line of the two identified rectangles based on the pixel coordinates of the multiple center points to calculate the depth of each center point; calculating the coordinates of each center point based on the depth of each center point to obtain the position; and calculating the yaw angle based on the coordinates.

[0009] Optionally, in one embodiment of the present application, the calculation formula of the coordinates is:

[0010] X=(uc x )Z / f x ,

[0011] Y=(vc y )Z / f y ,

[0012] Z=Z,

[0013] Where X is the X-axis coordinate of the pixel point 3D coordinate, Y is the Y-axis coordinate of the pixel point 3D coordinate, Z is the Z-axis coordinate of the pixel point 3D coordinate, u is the x-axis coordinate of the pixel point image coordinate system, and v is the y-axis coordinate of the pixel point image coordinate system. (f x ,f y ) is the camera’s intrinsic focal length and (c x ,c y ) principal point coordinates, x and y are the x-axis and y-axis of the camera intrinsic parameters respectively.

[0014] Optionally, in one embodiment of the present application, the calculation formula of the yaw angle is:

[0015]

[0016] Among them, β is the yaw angle, X VN , Y VN is the center coordinate of the square in the local coordinates between the camera and the following vehicle, and N is the number of observation data points involved in the fitting process when tracking the leading and following vehicles.

[0017] The second aspect of the present application provides a monocular camera-based intelligent connected bus multi-vehicle formation perception device, including: a detection module, used to detect a detection frame of a front vehicle in a multi-vehicle formation of an intelligent connected bus on an image; a cropping module, used to crop out the back panel of the front vehicle based on the detection frame to detect multiple center points of the front vehicle; a generation module, used to calculate the position and yaw angle of the front vehicle in the camera coordinate system according to the multiple center points, and convert the coordinates of the front vehicle in the camera coordinate system to the vehicle coordinate system of the rear vehicle according to the external parameter matrix obtained by calibrating the camera and the rear vehicle to obtain the coordinates of the front vehicle in the vehicle coordinate system of the rear vehicle, and generate the intelligent connected bus multi-vehicle formation perception result.

[0018] Optionally, in one embodiment of the present application, the cropping module includes: an acquisition unit, used to acquire a picture of the back panel and the position of the back panel in the picture; a processing unit, used to binarize the picture to obtain a binarization result; and a determination unit, used to identify the vehicle contour based on the binarization result, and determine the multiple center points based on the vehicle contour.

[0019] Optionally, in one embodiment of the present application, the generation module includes: a first calculation unit, used to calculate the pixel distance from the coordinates of each center point to the horizontal straight line of the two rectangles identified according to the pixel coordinates of the multiple center points, so as to calculate the depth of each center point; a second calculation unit, used to calculate the coordinates of each center point according to the depth of each center point to obtain the position; and a third calculation unit, used to calculate the yaw angle according to the coordinates.

[0020] Optionally, in one embodiment of the present application, the calculation formula of the coordinates is:

[0021] X=(uc x )Z / f x ,

[0022] Y=(vc y )Z / f y ,

[0023] Z=Z,

[0024] Where X is the X-axis coordinate of the pixel point 3D coordinate, Y is the Y-axis coordinate of the pixel point 3D coordinate, Z is the Z-axis coordinate of the pixel point 3D coordinate, u is the x-axis coordinate of the pixel point image coordinate system, and v is the y-axis coordinate of the pixel point image coordinate system. (f x ,f y ) is the camera’s intrinsic focal length and (c x ,c y ) principal point coordinates, x and y are the x-axis and y-axis of the camera intrinsic parameters respectively.

[0025] Optionally, in one embodiment of the present application, the calculation formula of the yaw angle is:

[0026]

[0027] Among them, β is the yaw angle, X VN , Y VN is the center coordinate of the square in the local coordinates between the camera and the following vehicle, and N is the number of observation data points involved in the fitting process when tracking the leading and following vehicles.

[0028] The third aspect of the present application provides a vehicle, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for sensing multi-vehicle formation of intelligent connected buses based on a monocular camera as described in the above embodiment.

[0029] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned monocular camera-based intelligent connected bus multi-vehicle formation perception method.

[0030] The fifth aspect of the present application provides a computer program product, which, when executed, is used to implement the above-mentioned monocular camera-based intelligent connected bus multi-vehicle formation perception method.

[0031] The embodiment of the present application can detect the detection frame of the front vehicle in the multi-vehicle formation of the intelligent networked bus on the image, and cut out the back plate of the front vehicle, so as to calculate the position and yaw angle of the front vehicle in the camera coordinate system by detecting multiple center points of the front vehicle, and convert the coordinates of the front vehicle in the camera coordinate system to the vehicle coordinate system of the rear vehicle according to the external parameter matrix obtained by calibrating the camera and the rear vehicle, so as to obtain the coordinates of the front vehicle in the vehicle coordinate system of the rear vehicle, thereby generating the perception result of the multi-vehicle formation of the intelligent networked bus, which can effectively identify the position of each vehicle in the fleet and its relative relationship, realize the accurate perception of the vehicle formation driving, and reduce the system cost and complexity. Thus, it solves the problems in the related technology that the overall deployment cost of the vehicle formation technology may increase and affect its performance and accuracy due to the relatively high cost of the laser radar and radar systems and their sensitivity to environmental conditions, and the system complexity and maintenance difficulty are increased due to the need for precise calibration and synchronization of the multi-camera system.

[0032] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A flowchart of a method for sensing a multi-vehicle formation of an intelligent connected bus based on a monocular camera according to an embodiment of the present application;

[0035] Figure 2 This is a flow chart of a method for sensing multi-vehicle formation of intelligent networked buses based on a monocular camera according to an embodiment of the present application;

[0036] Figure 3 is a flowchart of camera coordinate system calculation according to one embodiment of the present application;

[0037] Figure 4 This is a schematic diagram of a back panel of a trolley according to an embodiment of the present application;

[0038] Figure 5 This is a schematic diagram of a car after binarization processing according to an embodiment of the present application;

[0039] Figure 6 A schematic diagram of contour detection according to an embodiment of the present application;

[0040] Figure 7 A schematic diagram of detecting straight line classification according to an embodiment of the present application;

[0041] Figure 8A schematic diagram of the intersection of a horizontal straight line and a vertical straight line according to an embodiment of the present application;

[0042] Fig. 9 A schematic diagram of a dual-threshold detection interval according to an embodiment of the present application;

[0043] Fig.10 A schematic diagram of the Euclidean distance between a preceding vehicle and the vehicle according to an embodiment of the present application;

[0044] Fig.11 A schematic diagram of the relative positions of a front vehicle and a rear vehicle according to an embodiment of the present application;

[0045] Fig.12 A schematic diagram of a camera marking system according to an embodiment of the present application;

[0046] Fig.13 A schematic diagram of a model for counteracting a rolling effect of a vehicle according to an embodiment of the present application;

[0047] Fig.14 A schematic diagram of camera coordinate system imaging according to an embodiment of the present application;

[0048] Fig.15 A schematic diagram of coordinate system conversion according to an embodiment of the present application;

[0049] Fig.16 A schematic diagram of multi-vehicle formation perception of intelligent connected buses based on a monocular camera according to an embodiment of the present application;

[0050] Fig.17 It is a schematic diagram of the structure of a vehicle provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0052] The following describes a method and device for sensing a multi-vehicle formation of an intelligent connected bus based on a monocular camera in accordance with an embodiment of the present application with reference to the accompanying drawings. In view of the related technologies mentioned in the above background technology center, since the cost of lidar and radar systems is relatively high and they are sensitive to environmental conditions, the overall deployment cost of vehicle platooning technology may increase and affect its performance and accuracy. In addition, since the multi-camera system requires precise calibration and synchronization, the complexity and maintenance difficulty of the system are increased. The present application provides a method for sensing multi-vehicle platooning of intelligent connected buses based on a monocular camera. In this method, the detection frame of the front vehicle in the multi-vehicle platoon of the intelligent connected bus can be detected on the image, and the back panel of the front vehicle can be cropped out to calculate the position and yaw angle of the front vehicle in the camera coordinate system by detecting multiple center points of the front vehicle. The coordinates of the front vehicle in the camera coordinate system are converted to the vehicle coordinate system of the rear vehicle according to the external parameter matrix obtained by calibrating the camera and the rear vehicle to obtain the coordinates of the front vehicle in the vehicle coordinate system of the rear vehicle, thereby generating the sensing result of the multi-vehicle platoon of the intelligent connected bus, which can effectively identify the position of each vehicle in the fleet and their relative relationship, realize accurate perception of vehicle platoon driving, and reduce system cost and complexity. This solves the problems in the related technologies, such as the relatively high cost of lidar and radar systems and their sensitivity to environmental conditions, which may increase the overall deployment cost of vehicle platooning technology and affect its performance and accuracy, and the need for precise calibration and synchronization of multi-camera systems, which increases the complexity of the system and the difficulty of maintenance.

[0053] Specifically, Figure 1 A flow chart of a method for sensing multi-vehicle platooning of intelligent connected buses based on a monocular camera provided in an embodiment of the present application.

[0054] like Figure 1 As shown, the method for sensing multiple vehicles in a platoon of intelligent connected buses based on a monocular camera includes the following steps:

[0055] In step S101, a detection frame of a front vehicle in a multi-vehicle formation of an intelligent network-connected bus is detected on an image.

[0056] Specifically, combined Figure 2 As shown, the embodiment of the present application can detect the detection frame of the front vehicle in the multi-vehicle formation of the intelligent network-connected bus on the image, which specifically includes:

[0057] Step S201: receiving a monocular camera image output by a rear vehicle;

[0058] Step S202: performing brightness equalization processing on the received camera image;

[0059] Step S203: Obtain the detection frame of the front vehicle on the image through the post-fusion module, that is, the real coordinates (x, y, z, theta (yaw angle)) of the front vehicle;

[0060] Step S204: Determine whether the detection fails based on the real coordinates of the front vehicle. If the yaw angle is not between [-15, 15] degrees, the detection fails. Otherwise, the target frame size of the front vehicle's back panel and the pixel coordinates (x, y, w, h) of the back panel in the image can be calculated based on the front vehicle's actual physical size and the camera's internal and external parameters.

[0061] The embodiment of the present application can effectively enhance the vehicle perception capability, improve the accuracy of detection and positioning, and provide an information basis for subsequent control by detecting the detection frame of the front vehicle in the multi-vehicle formation of the intelligent connected bus on the image.

[0062] In step S102, based on the detection frame, the back panel of the front vehicle is cropped out to detect multiple center points of the front vehicle.

[0063] By cutting out the back panel of the front vehicle, the embodiment of the present application can effectively detect multiple center points of the front vehicle according to the shapes and sizes of different vehicles, which helps to accurately understand the position and shape of the front vehicle, improve the accuracy and flexibility of vehicle perception, and provide more reliable data support for subsequent control, recognition and other functions.

[0064] Optionally, in one embodiment of the present application, the back panel of the front vehicle is cropped out based on the detection frame to detect multiple center points of the front vehicle, including: obtaining a picture of the back panel and the position of the back panel in the picture; binarizing the picture to obtain a binarization result; identifying the vehicle outline based on the binarization result, and determining multiple center points based on the vehicle outline.

[0065] In the actual implementation process, combined with Figure 2 As shown, in order to reduce the influence of the image background, the embodiment of the present application can cut out the back panel of the front vehicle based on the detection frame to detect multiple center points of the front vehicle. Specifically, a model can be established based on the cut-out back panel of the front vehicle, which specifically includes:

[0066] Step S205: Based on the detection frame and the camera image after brightness equalization processing, the front vehicle back panel image can be cropped.

[0067] Step S206: Calculate the camera coordinate system position. The calculation steps are as follows:

[0068] like Figure 3 As shown, step S301: Figure 4 As shown, obtain the image of the front vehicle back panel and the position of the back panel in the original image;

[0069] Step S302: Figure 5 As shown, the image of the front vehicle back panel is binarized and binarized after applying Canny edge detection;

[0070] Step S303: The vehicle backboard image is converted into a black and white image after binarization, where the white area of ​​the original image is black and the other colors are white;

[0071] Step S304: Figure 6 As shown, the Canny edge detection algorithm is applied to identify the vehicle contour;

[0072] Step S305: Determine whether the area of ​​the contour is greater than 20% of the total area of ​​the image and whether the contour area is greater than 2. If so, draw the contour on a blank canvas, otherwise the detection fails. Contour detection includes the following steps:

[0073] a) Remove noise and use Gaussian filter to remove image noise and retain the overall structure of the image. For a pixel at position (m,n), its gray value is f(m,n), and the gray value after Gaussian filtering will become:

[0074]

[0075] Among them, g σ (m,n) is the function of the Gaussian filter; (m,n) is the coordinate of a pixel; f(m,n) is the grayscale value; σ is the standard deviation of the Gaussian filter.

[0076] b) Calculate the gradient strength and direction: Using the Sobel operator to calculate the gradient and direction of the image intensity, the edge strength and direction in the image can be found. The direction of the gradient is always perpendicular to the edge, g x (m,n) and g y (m,n) are the gradient values ​​in different directions. The formulas for gradient value and gradient direction can be:

[0077]

[0078]

[0079] Among them, G(m,n) is the gradient strength function; θ is the gradient direction; g x (m,n),g y (m,n) are the gradients of the image in the horizontal and vertical directions respectively.

[0080] c) Non-maximum suppression: Obtain refined edges and retain only the key parts of the edge. On each possible edge, observe whether this point is the local maximum along the gradient direction in its neighborhood. If so, retain it, otherwise suppress it.

[0081]

[0082] Among them, M T(m,n) is the pixel value of the processed image at position (m,n); M(m,n) is the pixel value of the original image at position (m,n); T is the preset threshold.

[0083] d) Double threshold detection: determine which edges are real edges. Figure 7 As shown in the figure, by setting the maximum threshold and the minimum threshold, if the intensity of a pixel position exceeds the maximum threshold, it is judged to be an edge; if the intensity of a pixel position is lower than the minimum threshold, it is judged to be not an edge; if the intensity of a pixel position is between the two thresholds, it can be judged to be an edge if and only if it is connected to a "determined edge" pixel.

[0084] e) Edge connection: Analyze the edge between two thresholds to see if it is connected to the edge pixel. If so, it is also judged as an edge, thus obtaining the final edge image.

[0085] Step S306: Figure 8 and Fig. 9 As shown, the binary image with contour patterns is subjected to line detection (Hough Transform). The detected lines are classified (approximately horizontal lines and vertical lines) and converted to the polar coordinate system. At this time, because the detected lines are segmented, the lines on the same side are merged into one line through similarity detection, and the intersection of the horizontal and vertical lines is calculated to merge adjacent points. The three center pixel points (u1, v1), (u2, v2), and (u3, v3) on the two rectangular plates of the car can be found. Among them, the center pixel point is the 3D coordinate of the front car in the camera coordinate system, and the remaining points assist in calculating the yaw angle of the front car. The specific formula can be as follows:

[0086] ρ=xcosθ+ysinθ

[0087] Where ρ is the distance from the origin to the line, which can be positive or negative, depending on the position of the line and its relative orientation to the origin; θ is the angle between the normal of the line and the positive direction of the x-axis, which usually ranges from 0 to π.

[0088] In polar coordinates, the two straight lines can be expressed as r1 = xcosθ1 + ysinθ1 and r2 = xcosθ2 + ysinθ2. To calculate the intersection of the two straight lines, the values ​​of x and y can be obtained by solving the system of equations, as shown below:

[0089]

[0090]

[0091] Among them, x is the horizontal coordinate of the intersection of the two straight lines, y is the vertical coordinate of the intersection of the two straight lines, θ1 and θ2 are the angles between the two straight lines and the x-axis, and r1 and r2 are the radial distances in polar coordinates, corresponding to the two straight lines respectively.

[0092] The embodiment of the present application can obtain a picture of the back panel and the position of the back panel in the picture, which is helpful for further analysis and processing of specific areas of the vehicle. By binarizing the back panel image, the outline of the vehicle can be highlighted to better identify the vehicle shape and boundaries, providing basic data for subsequent feature extraction and tracking, and improving the accuracy and completeness of vehicle perception.

[0093] In step S103, the position and yaw angle of the leading vehicle in the camera coordinate system are calculated based on multiple center points, and the coordinates of the leading vehicle in the camera coordinate system are converted to the vehicle coordinate system of the rear vehicle based on the external parameter matrix obtained by calibrating the camera and the rear vehicle, so as to obtain the coordinates of the leading vehicle in the vehicle coordinate system of the rear vehicle, and generate the perception result of the multi-vehicle formation of the intelligent connected bus.

[0094] Specifically, combined Figure 2 As shown, the embodiment of the present application can calculate the position and yaw angle of the front vehicle in the camera coordinate system according to multiple center points, and convert the coordinates of the front vehicle in the camera coordinate system to the vehicle coordinate system of the rear vehicle according to the external parameter matrix obtained by calibrating the camera and the rear vehicle, so as to obtain the coordinates of the front vehicle in the vehicle coordinate system of the rear vehicle, wherein the specific steps are as follows:

[0095] Step S207: Calculate the position and yaw angle of the front vehicle in the camera coordinate system according to the multiple center points;

[0096] Step S208: converting the coordinates of the front vehicle in the camera coordinate system to the laser radar coordinate system according to the external parameters of the camera and the laser radar;

[0097] Step S209: converting the front vehicle in the laser radar coordinate system into the front vehicle front axle coordinate system according to the external parameters of the laser radar and the front axle;

[0098] Step S210: Obtain coordinates based on the front axle coordinate system of the front vehicle.

[0099] The embodiment of the present application can accurately determine the position and posture of the leading vehicle relative to the camera by calculating the positions and yaw angles of multiple center points, and with the help of the external parameter matrix of the camera and the rear vehicle, the position and posture information of the leading vehicle in the camera coordinate system can be converted to the vehicle coordinate system of the rear vehicle, which can effectively improve the accuracy and reliability of the perception results of the multi-vehicle formation of intelligent connected buses, and help to improve the safety and coordination between vehicles.

[0100] Optionally, in one embodiment of the present application, the position and yaw angle of the leading vehicle in the camera coordinate system are calculated based on multiple center points, including: calculating the pixel distance from the coordinates of each center point to the horizontal straight line of the two identified rectangles based on the pixel coordinates of the multiple center points to calculate the depth of each center point; calculating the coordinates of each center point based on the depth of each center point to obtain the position; and calculating the yaw angle based on the coordinates.

[0101] Specifically, combined Fig.10 As shown, the embodiment of the present application can calculate the position and yaw angle of the camera coordinate system of the front vehicle according to the three detected center points, wherein, according to the pixel coordinates of the three midpoints, the pixel distances h1, h2, and h3 (unit: pixel) from the coordinates of the three center pixel points to the horizontal straight lines of the two identified rectangles are calculated respectively, that is, the pixel height, and the physical world height h (unit: m) of the front door is obtained. Fig.11 As shown, the depths Z1, Z2, and Z3 of the three central pixels can be calculated.

[0102] d=h / (y-yo) / f

[0103] Among them, d is the distance between the front vehicle and the vehicle; h is the height of the camera from the ground; f is the focal length; y is the ordinate of the coordinate position mapped to the vehicle; yo is the ordinate of the center coordinate of the imaging plane.

[0104] Based on the depth information of a given pixel, the 3D coordinates of the point in the camera coordinate system can be calculated using the camera's intrinsic parameters and depth value. The pixel coordinates are converted to camera-centered coordinates using the inverse matrix of the camera's intrinsic parameters, and then scaled according to the depth value. The camera's intrinsic parameters include the focal length (f x , f y ) and the principal point coordinates (c x 、c y ), for a pixel point (u, v) with depth (Z), the 3D coordinate calculation formula can be as follows:

[0105] X=(uc x )Z / f x

[0106] Y=(vc y )Z / f y

[0107] Z=Z

[0108] Where X is the X-axis coordinate of the pixel point 3D coordinate, Y is the Y-axis coordinate of the pixel point 3D coordinate, Z is the Z-axis coordinate of the pixel point 3D coordinate, u is the x-axis coordinate of the pixel point image coordinate system, and v is the y-axis coordinate of the pixel point image coordinate system. (f x ,f y) is the camera’s intrinsic focal length and (c x ,c y ) principal point coordinates, x and y are the x-axis and y-axis of the camera intrinsic parameters respectively.

[0109] By converting the 2D pixel coordinates with depth information into 3D space coordinates relative to the camera, the three center pixel points are converted to the camera coordinate system to obtain (X1, Y1, Z1), (X2, Y2, Z2), and (X3, Y3, Z3), where the middle point (X2, Y2, Z2) represents the coordinates of the front vehicle in the camera coordinate system.

[0110] Furthermore, combined with Fig.12 As shown, in the embodiment of the present application, in order for the front vehicle to always see the rear vehicle, the camera can select a certain angle so that the logo of the front vehicle always appears in the FOV of the rear vehicle camera, so α=α En +α s , where α En is the rotation angle of the camera, α s is the angle from the center of the middle square to the optical axis of the camera, α is the deflection of the front vehicle relative to the center line of the rear vehicle (relative to the deflection of the camera), if α assumes that the camera is consistent with the center line of the rear vehicle, that is, the camera rotation is 0.

[0111] In addition, combined Figures 13 to 15 As shown, the embodiment of the present application can calculate the yaw angle of the front vehicle according to the points in the three camera coordinate systems. The calculation of the yaw angle can actually be regarded as a top view. Due to the possible bumps of the vehicle, the vehicle camera has a deflection of γ on the Z axis of the camera coordinate system, where the calculation formula of γ can be as follows:

[0112]

[0113] Where γ is the inclination angle relative to the horizontal plane, x cn ,y cn is the position coordinate in the camera coordinate system, c is the camera, and n is the center pixel point.

[0114] The embodiment of the present application projects the point to the horizontal coordinate system, and can construct a conversion model of the center pixel point in the camera coordinate system with γ deflection on the z-axis and the center pixel point in the horizontal camera coordinate system, wherein the model can be as follows:

[0115] X CN =X HN cosγ-Y HN sinγ

[0116] Z cN =Z HN

[0117] Among them, XCN , Z CN is the position coordinate in the camera coordinate system, X HN , Y HN , Z HN is the coordinate of the center of the square relative to the horizontal plane, γ is the inclination angle relative to the horizontal plane, H is the horizontal level, and N is the number of central pixels.

[0118] Since the relative position between the front and rear vehicles corresponds only to the XZ coordinates, the y-axis can be ignored and only the X-axis needs to be projected horizontally:

[0119]

[0120] Finally, the coordinates are converted into a top view to calculate β, that is, the yaw angle. The relative angle β between the front vehicle and the rear vehicle can be calculated as follows:

[0121]

[0122] Among them, β is the yaw angle of the front vehicle relative to the rear vehicle. Assume that X VN =X CN , Y VN =Z CN , V stands for vertical, which means converting xz into xy. VN , Y VN is the center coordinate of the square in the local coordinates between the camera and the following vehicle, and N is the number of observation data points involved in the fitting process when tracking the leading and following vehicles.

[0123] The embodiments of the present application help to further improve the perception results of the leading vehicle by calculating the depth, coordinates and yaw angles of multiple center points, and provide more accurate and reliable data support for functions such as multi-vehicle formation, vehicle distance control and collaborative driving of intelligent connected buses, thereby improving overall safety and efficiency.

[0124] Optionally, in one embodiment of the present application, the coordinate calculation formula is:

[0125] X=(uc x )Z / f x ,

[0126] Y=(vc y )Z / f y ,

[0127] Z=Z,

[0128] Where X is the X-axis coordinate of the pixel point 3D coordinate, Y is the Y-axis coordinate of the pixel point 3D coordinate, Z is the Z-axis coordinate of the pixel point 3D coordinate, u is the x-axis coordinate of the pixel point image coordinate system, and v is the y-axis coordinate of the pixel point image coordinate system. (f x ,f y ) is the camera’s intrinsic focal length and (c x ,c y ) principal point coordinates, x and y are the x-axis and y-axis of the camera intrinsic parameters respectively.

[0129] The embodiment of the present application can calculate the pixel distance from the coordinates of each center point to the horizontal straight line of the two rectangles identified according to the pixel coordinates of multiple center points, so as to calculate the depth of each center point and calculate the coordinates of each center point to obtain the position, which is beneficial to improve the reliability and accuracy of the coordinates, thereby enhancing the coordination and safety of the intelligent connected bus.

[0130] Optionally, in one embodiment of the present application, the calculation formula of the yaw angle is:

[0131]

[0132] Among them, β is the yaw angle of the front vehicle relative to the rear vehicle, X VN , Y VN is the center coordinate of the square in the local coordinates between the camera and the following vehicle, and N is the number of observation data points involved in the fitting process when tracking the leading and following vehicles.

[0133] By calculating the yaw angle, the embodiment of the present application can accurately understand the vehicle's direction and posture, provide key information for real-time monitoring of the vehicle's driving status, and help to rationally plan the driving route, improve traffic efficiency, and improve the safety of vehicle driving.

[0134] According to the detection frame of the front vehicle in the multi-vehicle formation of the intelligent networked bus proposed in the embodiment of the present application, the back panel of the front vehicle can be cut out to calculate the position and yaw angle of the front vehicle in the camera coordinate system by detecting multiple center points of the front vehicle, and the coordinates of the front vehicle in the camera coordinate system are converted to the vehicle coordinate system of the rear vehicle according to the external parameter matrix obtained by calibrating the camera and the rear vehicle to obtain the coordinates of the front vehicle in the vehicle coordinate system of the rear vehicle, thereby generating the perception result of the multi-vehicle formation of the intelligent networked bus, which can effectively identify the position of each vehicle in the fleet and its relative relationship, realize accurate perception of the vehicle formation driving, and reduce the system cost and complexity. Thus, the problem in the related technology that the cost of the laser radar and radar systems is relatively high and sensitive to environmental conditions, which may increase the overall deployment cost of the vehicle formation technology and affect its performance and accuracy, and that the multi-camera system requires precise calibration and synchronization, which increases the complexity of the system and the difficulty of maintenance.

[0135] Next, a monocular camera-based intelligent connected bus multi-vehicle formation perception device proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0136] Fig.16 It is a structural schematic diagram of a monocular camera-based intelligent connected bus multi-vehicle formation perception device according to an embodiment of the present application.

[0137] like Fig.16 As shown, the intelligent connected bus multi-vehicle formation perception device 10 based on a monocular camera includes: a detection module 100, a cropping module 200 and a generation module 300.

[0138] Specifically, the detection module 100 is used to detect the detection frame of the front vehicle in the multi-vehicle formation of the intelligent network-connected bus on the image;

[0139] A cropping module 200, for cropping the back panel of the front vehicle based on the detection frame to detect multiple center points of the front vehicle;

[0140] The generation module 300 is used to calculate the position and yaw angle of the leading vehicle in the camera coordinate system based on multiple center points, and to convert the coordinates of the leading vehicle in the camera coordinate system to the vehicle coordinate system of the rear vehicle based on the external parameter matrix obtained by calibrating the camera and the rear vehicle, so as to obtain the coordinates of the leading vehicle in the vehicle coordinate system of the rear vehicle, and generate the perception results of the multi-vehicle formation of the intelligent connected bus.

[0141] Optionally, in one embodiment of the present application, the cropping module 200 includes: an acquisition unit, a processing unit and a determination unit.

[0142] The acquisition unit is used to acquire the image of the backplane and the position of the backplane in the image;

[0143] A processing unit, used for performing binarization processing on the image to obtain a binarization result;

[0144] The determination unit is used to identify the vehicle contour according to the binarization result and determine multiple center points according to the vehicle contour.

[0145] Optionally, in one embodiment of the present application, the generation module 300 includes: a first calculation unit, a second calculation unit and a third calculation unit.

[0146] The first calculation unit is used to calculate the pixel distance from the coordinates of each center point to the horizontal straight line identifying the two rectangles according to the pixel coordinates of the multiple center points, so as to calculate the depth of each center point;

[0147] A second calculation unit is used to calculate the coordinates of each center point according to the depth of each center point to obtain a position;

[0148] The third calculation unit is used to calculate the yaw angle according to the coordinates.

[0149] Optionally, in one embodiment of the present application, the coordinate calculation formula is:

[0150] X=(uc x )Z / f x ,

[0151] Y=(vc y )Z / f y ,

[0152] Z=Z,

[0153] Where X is the X-axis coordinate of the pixel point 3D coordinate, Y is the Y-axis coordinate of the pixel point 3D coordinate, Z is the Z-axis coordinate of the pixel point 3D coordinate, u is the x-axis coordinate of the pixel point image coordinate system, and v is the y-axis coordinate of the pixel point image coordinate system. (f x ,f y ) is the camera’s intrinsic focal length and (c x ,c y ) principal point coordinates, x and y are the x-axis and y-axis of the camera intrinsic parameters respectively.

[0154] Optionally, in one embodiment of the present application, the calculation formula of the yaw angle is:

[0155]

[0156] Among them, β is the yaw angle of the front vehicle relative to the rear vehicle, X VN , Y VN is the center coordinate of the square in the local coordinates between the camera and the following vehicle, and N is the number of observation data points involved in the fitting process when tracking the leading and following vehicles.

[0157] It should be noted that the aforementioned explanation of the embodiment of the method for sensing multi-vehicle platooning of intelligent connected buses based on a monocular camera is also applicable to the device for sensing multi-vehicle platooning of intelligent connected buses based on a monocular camera of this embodiment, and will not be repeated here.

[0158] According to the intelligent networked bus multi-vehicle formation perception device based on a monocular camera proposed in the embodiment of the present application, the detection frame of the front vehicle in the multi-vehicle formation of the intelligent networked bus can be detected on the image, and the back plate of the front vehicle can be cropped out to calculate the position and yaw angle of the front vehicle in the camera coordinate system by detecting multiple center points of the front vehicle, and the coordinates of the front vehicle in the camera coordinate system are converted to the vehicle coordinate system of the rear vehicle according to the external parameter matrix obtained by calibrating the camera and the rear vehicle, so as to obtain the coordinates of the front vehicle in the vehicle coordinate system of the rear vehicle, thereby generating the intelligent networked bus multi-vehicle formation perception result, which can effectively identify the position of each vehicle in the fleet and its relative relationship, realize accurate perception of vehicle formation driving, and reduce system cost and complexity. Thus, the problem in the related technology that the cost of laser radar and radar systems is relatively high and sensitive to environmental conditions, which may increase the overall deployment cost of vehicle formation technology and affect its performance and accuracy, and that the multi-camera system requires precise calibration and synchronization, which increases the complexity of the system and the difficulty of maintenance.

[0159] Fig.17 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application. The vehicle may include:

[0160] A memory 1701 , a processor 1702 , and a computer program stored in the memory 1701 and executable on the processor 1702 .

[0161] When the processor 1702 executes the program, the method for sensing multi-vehicle formation of intelligent connected buses based on a monocular camera provided in the above embodiment is implemented.

[0162] Furthermore, the vehicle also includes:

[0163] The communication interface 1703 is used for communication between the memory 1701 and the processor 1702 .

[0164] The memory 1701 is used to store computer programs that can be executed on the processor 1702 .

[0165] The memory 1701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0166] If the memory 1701, the processor 1702 and the communication interface 1703 are implemented independently, the communication interface 1703, the memory 1701 and the processor 1702 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.17 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0167] Optionally, in a specific implementation, if the memory 1701, the processor 1702 and the communication interface 1703 are integrated on a chip, the memory 1701, the processor 1702 and the communication interface 1703 can communicate with each other through an internal interface.

[0168] The processor 1702 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0169] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for sensing multi-vehicle formation of intelligent connected buses based on a monocular camera.

[0170] An embodiment of the present application also provides a computer program product, which can run computer instructions. When the computer instructions are executed by a processor, the above-mentioned method for perceiving multi-vehicle formations of intelligent connected buses based on a monocular camera is implemented.

[0171] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0172] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0173] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0174] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0175] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0176] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0178] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for sensing multi-vehicle formation of intelligent networked buses based on a monocular camera, characterized in that: The following steps are involved: Detect the detection frame of the front vehicle in the multi-vehicle formation of intelligent connected buses on the image; Based on the detection frame, cutting out the back panel of the front vehicle to detect multiple center points of the front vehicle; The position and yaw angle of the leading vehicle in the camera coordinate system are calculated according to the multiple center points, and the coordinates of the leading vehicle in the camera coordinate system are converted to the vehicle coordinate system of the trailing vehicle according to the extrinsic parameter matrix obtained by calibrating the camera and the trailing vehicle, so as to obtain the coordinates of the leading vehicle in the vehicle coordinate system of the trailing vehicle, and the coordinates in the vehicle coordinate system of the trailing vehicle are converted to the front axle coordinate system of the leading vehicle through the preset extrinsic parameters of the trailing vehicle and the front axle, so as to generate the perception result of the multi-vehicle formation of the intelligent connected bus through the coordinates in the front axle coordinate system of the leading vehicle; The calculating the position and the yaw angle of the front vehicle in the camera coordinate system according to the multiple center points includes: Calculating the pixel distance from the coordinates of each center point to the horizontal straight line identifying the two rectangles according to the pixel coordinates of the multiple center points, so as to calculate the depth of each center point; Calculate the coordinates of each center point according to the depth of each center point to obtain the position; The tilt angle relative to the horizontal plane is calculated according to the position in the camera coordinate system, and the center coordinates of the square between the camera and the rear vehicle are determined by the tilt angle, so as to calculate the yaw angle based on the center coordinates of the square.

2. The method according to claim 1, characterized in that The step of cutting out the back panel of the front vehicle based on the detection frame to detect multiple center points of the front vehicle includes: Obtaining a picture of the backplane and a position of the backplane in the picture; Performing binarization processing on the image to obtain a binarization result; A vehicle outline is identified according to the binarization result, and the plurality of center points are determined according to the vehicle outline.

3. The method according to claim 1, characterized in that The coordinates are calculated as follows: X=(u-c x )Z / f x , Y=(v-c y )Z / f y , Z=Z, Where X is the X-axis coordinate of the pixel point 3D coordinate, Y is the Y-axis coordinate of the pixel point 3D coordinate, Z is the Z-axis coordinate of the pixel point 3D coordinate, u is the x-axis coordinate of the pixel point image coordinate system, and v is the y-axis coordinate of the pixel point image coordinate system. (f x ,f y ) is the camera’s intrinsic focal length and (c x ,c y ) principal point coordinates, x and y are the x-axis and y-axis of the camera intrinsic parameters respectively.

4. The method according to claim 3, characterized in that The calculation formula of the yaw angle is: Among them, β is the yaw angle, X VN , Y VN is the center coordinate of the square in the local coordinates between the camera and the following vehicle, and N is the number of observation data points involved in the fitting process when tracking the leading and following vehicles.

5. A multi-vehicle formation sensing device for intelligent networked public transportation based on a monocular camera, characterized in that: include: A detection module, used to detect the detection frame of the front vehicle in the multi-vehicle formation of the intelligent network-connected bus on the image; A cropping module, used for cropping the back panel of the preceding vehicle based on the detection frame to detect multiple center points of the preceding vehicle; A generation module, for calculating the position and yaw angle of the leading vehicle in the camera coordinate system according to the multiple center points, and converting the coordinates of the leading vehicle in the camera coordinate system to the vehicle coordinate system of the trailing vehicle according to the extrinsic parameter matrix obtained by calibrating the camera and the trailing vehicle, so as to obtain the coordinates of the leading vehicle in the vehicle coordinate system of the trailing vehicle, and converting the coordinates in the vehicle coordinate system of the trailing vehicle to the front axle coordinate system of the leading vehicle through the preset extrinsic parameters of the trailing vehicle and the front axle, so as to generate the perception result of the multi-vehicle formation of the intelligent connected bus through the coordinates in the front axle coordinate system of the leading vehicle; The generating module comprises: a first calculating unit, configured to calculate the pixel distance from the coordinates of each center point to the horizontal straight line identifying the two rectangles according to the pixel coordinates of the multiple center points, so as to calculate the depth of each center point; The second calculation unit is used to calculate the coordinates of each center point according to the depth of each center point to obtain the position; the third calculation unit is used to calculate the inclination angle relative to the horizontal plane according to the position in the camera coordinate system, and determine the square center coordinates between the camera and the rear vehicle through the inclination angle, so as to calculate the yaw angle based on the square center coordinates.

6. The device according to claim 5, characterized in that The cutting module comprises: An acquisition unit, used for acquiring a picture of the backplane and a position of the backplane in the picture; A processing unit, used for performing binarization processing on the image to obtain a binarization result; A determination unit is used to identify a vehicle contour according to the binarization result, and determine the multiple center points according to the vehicle contour.

7. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for sensing multi-vehicle formation of intelligent connected buses based on a monocular camera as described in any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for sensing multi-vehicle formation of intelligent connected buses based on a monocular camera as described in any one of claims 1 to 4.

9. A computer program product, characterized in that When the computer program is executed, it is used to implement the method for sensing multi-vehicle formation of intelligent connected buses based on a monocular camera as described in any one of claims 1 to 4.

Images