Vehicle centering driving method, device, equipment and storage medium

Abstract

The invention belongs to the field of automobile control technology, and discloses a vehicle centering driving method, device, equipment and storage medium; the method comprises: acquiring a road image of a vehicle, identifying a drivable area in the road image; performing image analysis on the drivable area to obtain a line midpoint of the drivable area and pixel coordinates of the line midpoint; converting the pixel coordinates based on a vehicle-mounted coordinate system to obtain vehicle-mounted coordinates; obtaining a steering angle of the vehicle according to the vehicle-mounted coordinates and the vehicle coordinates, and controlling the vehicle to travel to the line midpoint according to the steering angle of the vehicle; the invention acquires a road image during driving and analyzes the road image to obtain the line midpoint in the image, converts the line midpoint into coordinates in the vehicle-mounted coordinate system and calculates the steering angle of the vehicle to the line midpoint, and controls the vehicle to travel in the center based on the steering angle, so that the vehicle passes through the drivable area to determine a target point instead of a lane line, and can control the vehicle travel more stably and accurately.

Description

Technical Field

[0001] The present invention relates to the field of automobile control technology, and in particular to a vehicle centering driving method, device, equipment and storage medium. Background Art

[0002] At present, in order to provide navigation or driving advice for vehicles, a lane center guide line will be generated based on the driving route. The lane center guide line is a virtual path generated by the lane lines on the left and right sides. It provides a reference path for the autonomous driving and assisted driving systems to guide the vehicle to drive near the center of the lane.

[0003] However, the current generation of lane center guide lines requires accurate lane line detection perception, but lane lines may sometimes be missing and are often not accurate enough, resulting in the vehicle not being able to center well during driving, and there are problems with left and right offsets, which may lead to a bad driving experience from the driving position. Summary of the invention

[0004] The main purpose of the present invention is to provide a vehicle centering driving method, device, equipment and storage medium, aiming to solve the technical problem that the existing technology has poor centering driving effect for automatic driving based on lane lines.

[0005] To achieve the above object, the present invention provides a vehicle centering driving method, the method comprising the following steps:

[0006] Acquire a road image of the vehicle and identify a drivable area in the road image;

[0007] Performing image analysis on the drivable area to obtain the midpoint of the line in the drivable area and the pixel coordinates of the midpoint of the line;

[0008] Performing coordinate transformation on the pixel coordinates based on the vehicle-mounted coordinate system of the vehicle to obtain the vehicle-mounted coordinates of the midpoint of the line;

[0009] The steering angle of the vehicle is obtained according to the vehicle-mounted coordinates and the vehicle coordinates, and the vehicle is controlled to travel to the midpoint of the route according to the steering angle of the vehicle to achieve vehicle centering driving.

[0010] Optionally, performing image analysis on the drivable area to obtain a line midpoint of the drivable area and a pixel coordinate of the line midpoint includes:

[0011] Obtaining the longitudinal resolution, scale factor and vehicle speed of the drivable area;

[0012] Obtaining a preview distance of the drivable area according to the longitudinal resolution, the proportional coefficient, and the vehicle speed;

[0013] Segmenting the drivable area based on the preview distance to obtain segmented images;

[0014] The image midpoint of the segmented image is identified, and the line midpoint of the drivable area and the pixel coordinates of the line midpoint are obtained according to the image midpoint.

[0015] Optionally, the identifying the image midpoint of the segmented image and obtaining the line midpoint of the drivable area and the pixel coordinates of the line midpoint according to the image midpoint includes:

[0016] Obtaining a reference pixel value of each pixel point of the segmented image;

[0017] Sorting the reference pixel points according to preset standard road surface pixel values ​​to obtain a reference pixel point set;

[0018] The midpoint of the line in the drivable area and the pixel coordinates of the midpoint of the line are obtained according to the pixel values ​​of the first pixel point and the last pixel point in the reference pixel point set.

[0019] Optionally, the performing coordinate conversion on the pixel coordinates based on a vehicle-mounted coordinate system of the vehicle to obtain the vehicle-mounted coordinates of the midpoint of the route includes:

[0020] Acquiring an imaging internal parameter of an image acquisition unit, wherein the image acquisition unit is used to acquire the road surface image;

[0021] Convert the pixel coordinates to an imaging coordinate system according to the imaging intrinsic reference to obtain a reference coordinate;

[0022] The reference coordinates are transformed according to the transformation matrix of the vehicle coordinate system to obtain the vehicle coordinates of the midpoint of the line.

[0023] Optionally, converting the pixel coordinates to an imaging coordinate system according to the imaging intrinsic reference to obtain a reference coordinate includes:

[0024] Obtaining the focal length and imaging depth of the image acquisition unit;

[0025] Determine an imaging coordinate system according to the imaging intrinsic parameters, and determine a pixel coordinate system according to the pixel coordinates;

[0026] Obtaining a projection relationship between the imaging coordinate system and the pixel coordinate system according to the focal length and the imaging depth;

[0027] Obtaining an imaging coordinate system conversion matrix according to the projection relationship and the imaging internal parameters;

[0028] The pixel coordinates are converted to an imaging coordinate system according to the imaging coordinate system conversion matrix to obtain reference coordinates.

[0029] Optionally, performing coordinate transformation on the reference coordinates according to a transformation matrix of a vehicle-mounted coordinate system to obtain the vehicle-mounted coordinates of the midpoint of the route includes:

[0030] Obtaining the intrinsic parameter matrix and extrinsic parameter matrix of the image acquisition unit;

[0031] Obtaining a transformation matrix of the imaging coordinate system and the vehicle coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix, wherein the transformation matrix includes a rotation transformation matrix and a translation transformation matrix;

[0032] The reference coordinates are transformed into a coordinate system according to the rotation transformation matrix and the translation transformation matrix to obtain the vehicle-mounted coordinates of the midpoint of the route.

[0033] Optionally, obtaining a steering angle of the vehicle according to the vehicle-mounted coordinates and the vehicle coordinates, and controlling the vehicle to travel to the midpoint of the route according to the steering angle of the vehicle to achieve vehicle centering driving, includes:

[0034] Obtaining the wheelbase of the vehicle;

[0035] Obtaining a relative relationship between the steering curvature of the vehicle and the front wheel turning angle according to the geometric kinematics model of the vehicle;

[0036] Obtaining a steering angle of the vehicle based on the relative relationship, the vehicle wheelbase, the vehicle-mounted coordinates, and the vehicle coordinates;

[0037] The vehicle is controlled to travel to the midpoint of the route according to the steering angle of the vehicle, so as to realize the centering driving of the vehicle.

[0038] In addition, to achieve the above-mentioned purpose, the present invention further proposes a vehicle centering driving device, the vehicle centering driving device comprising:

[0039] An image processing module, used to obtain a road image of the vehicle and identify a drivable area in the road image;

[0040] A coordinate conversion module, used for performing image analysis on the drivable area to obtain the line midpoint of the drivable area and the pixel coordinates of the line midpoint;

[0041] The coordinate conversion module is further used to perform coordinate conversion on the pixel coordinates based on the vehicle-mounted coordinate system of the vehicle to obtain the vehicle-mounted coordinates of the midpoint of the route;

[0042] The centering driving module is used to obtain the steering angle of the vehicle according to the vehicle-mounted coordinates and the vehicle coordinates, and control the vehicle to travel to the midpoint of the route according to the steering angle of the vehicle to achieve centering driving of the vehicle.

[0043] In addition, to achieve the above-mentioned purpose, the present invention also proposes a vehicle centering driving device, which includes: a memory, a processor, and a vehicle centering driving program stored in the memory and executable on the processor, and the vehicle centering driving program is configured to implement the steps of the vehicle centering driving method described above.

[0044] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a vehicle centering driving program is stored, and when the vehicle centering driving program is executed by a processor, the steps of the vehicle centering driving method described above are implemented.

[0045] The present invention acquires road surface images during driving and analyzes the road surface images to obtain the midpoint of the line in the image, converts the pixel coordinates of the midpoint of the line in the road surface image into coordinates in a vehicle-mounted coordinate system, obtains the steering angle of the vehicle when it travels to the midpoint of the line based on the coordinates in the vehicle-mounted coordinate system, controls the vehicle to travel in the center based on the steering angle, enables the vehicle to determine the target point instead of the lane line through the drivable area, can control the vehicle travel more stably, and can flexibly adjust the preview distance according to the vehicle speed, so that the vehicle can travel in the center with better effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a structural schematic diagram of a vehicle centering driving device in a hardware operating environment involved in an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a flow chart of a first embodiment of a vehicle centering driving method of the present invention;

[0048] Figure 3 A schematic diagram of a drivable area of ​​an embodiment of a vehicle centering driving method of the present invention;

[0049] Figure 4 A schematic diagram of the relative relationship between the steering curvature of the vehicle and the front wheel turning angle in an embodiment of the vehicle centering driving method of the present invention;

[0050] Figure 5 A schematic diagram of the relationship between the front wheel turning angle and the vehicle coordinates of an embodiment of the vehicle centering driving method of the present invention;

[0051] Figure 6 A schematic diagram of a flow chart of a second embodiment of a vehicle centering driving method of the present invention;

[0052] Figure 7 A schematic diagram of the conversion relationship between the pixel coordinate system and the imaging coordinate system of an embodiment of the vehicle centering driving method of the present invention;

[0053] Figure 8 A schematic diagram of the imaging coordinate system space of an embodiment of the vehicle centering driving method of the present invention;

[0054] Fig. 9 A schematic diagram of the conversion between the imaging coordinate system and the vehicle coordinate system of an embodiment of the vehicle centering driving method of the present invention;

[0055] Fig.10 It is a structural block diagram of the first embodiment of the vehicle centering driving device of the present invention.

[0056] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0058] Reference Figure 1 , Figure 1 A schematic diagram of the structure of a vehicle centering driving device in the hardware operating environment involved in an embodiment of the present invention.

[0059] like Figure 1 As shown, the vehicle centering driving device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0060] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the vehicle-centered driving device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0061] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a vehicle centering driving program.

[0062] exist Figure 1 In the vehicle centering driving device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the vehicle centering driving device of the present invention can be set in the vehicle centering driving device, and the vehicle centering driving device calls the vehicle centering driving program stored in the memory 1005 through the processor 1001, and executes the vehicle centering driving method provided by the embodiment of the present invention.

[0063] The embodiment of the present invention provides a vehicle centering driving method, referring to Figure 2 , Figure 2 It is a schematic flow chart of the first embodiment of the vehicle centering driving method of the present invention.

[0064] In this embodiment, the vehicle centering driving method includes the following steps:

[0065] Step S10: Acquire a road image of the vehicle and identify a drivable area in the road image.

[0066] It is understandable that this embodiment can be executed while the vehicle is driving, and the acquired road image can be acquired by the vehicle's image acquisition device (camera, imager, camera, etc.). It can be seen that the road image described here can be the road image directly in front of the vehicle.

[0067] It should be understood that the road surface image may include other backgrounds, obstacles, roads, etc. on both sides of the road when it is collected. Not the entire road surface image can be driven by the vehicle. It is necessary to analyze the road surface graphics to obtain the area in the road surface image that can be driven by the vehicle, that is, the driving area.

[0068] It should be noted that identifying the drivable area in the road image can be obtained by performing semantic segmentation on the road image. Image semantic segmentation is a computer vision task that aims to segment the input image into multiple regions and assign a semantic category label to each pixel to indicate which object or region in the image the pixel belongs to.

[0069] It should be further explained that semantic segmentation of road images can be performed by extracting features from road images through a trained neural network image segmentation model, classifying and assigning labels based on the extracted features, and segmenting each area in the road image based on the labels, thereby realizing the identification of drivable areas in the road image.

[0070] It should be noted that the executor of this embodiment is a vehicle centering driving device, wherein the vehicle centering driving device has functions such as data processing, data communication and program running. The vehicle centering driving device can be an integrated controller, a control computer and other devices, and of course it can also be other devices with similar functions, and this embodiment does not limit this.

[0071] Step S20: performing image analysis on the drivable area to obtain the line midpoint of the drivable area and the pixel coordinates of the line midpoint.

[0072] It is understandable that the end point of the route may be the midpoint of a straight line formed from the horizontal coordinate of the same pixel in the drivable area.

[0073] It should be understood that based on the collected road surface image, a pixel coordinate system is constructed in units of pixels with any pixel as the origin, and the pixel coordinates of the midpoint of the line are obtained according to the position of the midpoint of the line in the pixel coordinate system.

[0074] It should be noted that the image analysis of the drivable area to obtain the midpoint of the line of the drivable area and the pixel coordinates of the midpoint of the line includes: obtaining the longitudinal resolution, the proportional coefficient and the speed of the vehicle of the drivable area; obtaining the preview distance of the drivable area according to the longitudinal resolution, the proportional coefficient and the speed of the vehicle; dividing the drivable area based on the preview distance to obtain a segmented image; identifying the image midpoint of the segmented image, and obtaining the midpoint of the line of the drivable area and the pixel coordinates of the midpoint of the line according to the image midpoint.

[0075] It is understandable that the longitudinal resolution and proportional coefficient of the drivable area can be obtained by image parameters that have been set by the image acquisition device, and the vehicle speed can also be obtained by the engine speed of the vehicle.

[0076] It should be understood that the preview distance can be based on the distance between the vehicle and the midpoint of the line in the image captured by the image acquisition device. In a specific implementation, the identification of the drivable area can refer to Figure 3 The left side of the figure is the inner image. The area of ​​the car on the left and right in the inner image is the background area, the middle area is the drivable area, the vertical distance of the horizontal line on the drivable area is the preview distance, and the point in the middle of the horizontal line is the midpoint of the drivable area. The right side of the figure is a schematic diagram of the current driving situation of the vehicle.

[0077] It should be noted that the preview distance l is determined and the setting of the preview distance is related to the resolution of the image. At the same time, like human drivers, when the vehicle speed is high, human drivers will look farther to maintain the stability of the vehicle. The setting of the preview distance can refer to the following formula:

[0078]

[0079] Among them, height represents the vertical resolution of the image, n represents the scale factor, and v represents the vehicle speed.

[0080] The drivable area is segmented based on the preview distance, and the segmented image can be a straight line corresponding to the midpoint of the line where the preview distance is located. The segmentation is performed based on the straight line, and only the area where the straight line is located is retained for the next step of image analysis. In the specific implementation, reference can be made to Figure 3 , image segmentation can be Figure 3 The rest of the image except the horizontal line is cut off.

[0081] It should be further explained that the identifying of the image midpoint of the segmented image and obtaining the line midpoint of the drivable area and the pixel coordinates of the line midpoint based on the image midpoint include: obtaining reference pixel values ​​of each pixel of the segmented image; sorting the reference pixel points according to preset standard road surface pixel values ​​to obtain a reference pixel point set; and obtaining the line midpoint of the drivable area and the pixel coordinates of the line midpoint based on the pixel values ​​of the first pixel point and the last pixel point in the reference pixel point set.

[0082] It should be noted that the image is segmented at this distance, and the segmented image map_line consists of only one horizontal line. The drivable area is detected in the segmented image, and the built-in function numpy.where in the NumPy library of Opencv is used to obtain the pixel coordinate set whose pixel value is equal to the road surface label. Assuming that the RGB value of the drivable road surface in the semantic image is (128, 64, 128), the code for obtaining the road surface pixel point is:

[0083] List=numpy.where(map_line=[128,64,128])

[0084] It should be further explained that if the horizontal coordinates of all pixel points are continuous (the horizontal coordinate of the next point is 1 more than the previous point), it means that there is only one drivable area:

[0085] List={(x0, l), (x0+1, l), (x0+2, l),..., (x0+n, l)}

[0086] In this case, take the first and last points in the set to find the midpoint as the midpoint of the drivable area. The specific calculation method can refer to the following formula:

[0087]

[0088] Step S30: performing coordinate conversion on the pixel coordinates based on the vehicle-mounted coordinate system of the vehicle to obtain the vehicle-mounted coordinates of the midpoint of the line.

[0089] It is understandable that the vehicle-mounted coordinate system can be a world coordinate system, or a coordinate system constructed with the vehicle as the origin and the current driving direction as the coordinate axis. The vehicle-mounted coordinates of the midpoint of the route can convert the midpoint of the route in the road image to the coordinates under the vehicle-mounted coordinate system.

[0090] It should be understood that the coordinate transformation of the pixel coordinates based on the vehicle-mounted coordinate system may be to first transform the coordinates of the midpoint of the route in the pixel coordinate system into the coordinates in the imaging coordinate system of the image acquisition device, and then transform the coordinates of the midpoint of the route in the imaging coordinate system into the vehicle-mounted coordinates in the vehicle-mounted coordinate system.

[0091] Step S40: obtaining a steering angle of the vehicle according to the vehicle-mounted coordinates and the vehicle coordinates, and controlling the vehicle to travel to the midpoint of the route according to the steering angle of the vehicle to achieve vehicle centering.

[0092] It is understandable that the vehicle-mounted coordinates represent where the vehicle is going to travel. Driving the vehicle-mounted coordinates to the vehicle-mounted coordinates requires controlling the vehicle to perform operations such as execution and steering. Based on the vehicle-mounted coordinates and the vehicle-mounted coordinates, the steering angle required for the vehicle to travel to the vehicle-mounted coordinates can be calculated.

[0093] It should be noted that obtaining the steering angle of the self-vehicle according to the vehicle-mounted coordinates and the self-vehicle coordinates, and controlling the self-vehicle to travel to the midpoint of the route according to the steering angle of the self-vehicle to achieve vehicle centering driving, includes: obtaining the vehicle wheelbase of the self-vehicle; obtaining the relative relationship between the steering curvature of the self-vehicle and the front wheel turning angle according to the geometric kinematic model of the self-vehicle; obtaining the steering angle of the self-vehicle based on the relative relationship, the vehicle wheelbase, the vehicle-mounted coordinates and the self-vehicle coordinates; controlling the self-vehicle to travel to the midpoint of the route according to the steering angle of the self-vehicle to achieve vehicle centering driving.

[0094] It can be understood that the wheelbase of the vehicle is obtained; the relative relationship between the steering curvature and the front wheel angle of the vehicle is obtained according to the geometric kinematics model of the vehicle. Figure 4 Based on the vehicle coordinates in the figure, which can be expressed as (xw, yw), the vehicle geometric kinematics model is adopted, and the idea of ​​pure tracking algorithm is borrowed. By controlling the turning radius R of the vehicle so that the center control point of the rear axle of the vehicle reaches the midpoint of the line along the arc, the relationship between the steering curvature and the front wheel turning angle can be obtained:

[0095]

[0096] It should be noted that the steering angle of the vehicle can be obtained based on the relative relationship, the vehicle wheelbase, the vehicle coordinates and the vehicle coordinates by controlling the vehicle using the arc curvature from the vehicle to the midpoint of the route. Figure 5 The relationship between the front wheel steering angle and the vehicle coordinates can be obtained as follows:

[0097]

[0098] Where L is the vehicle wheelbase, which is a known value and is related to the vehicle shape, and δ is the final calculated steering angle.

[0099] This embodiment acquires road surface images during driving and analyzes the road surface images to obtain the midpoint of the line in the image, converts the pixel coordinates of the midpoint of the line in the road surface image into coordinates in the vehicle coordinate system, obtains the steering angle of the vehicle when it travels to the midpoint of the line based on the coordinates in the vehicle coordinate system, controls the vehicle to travel in the center based on the steering angle, enables the vehicle to determine the target point instead of the lane line through the drivable area, can control the vehicle travel more stably, and can flexibly adjust the preview distance according to the vehicle speed, so that the vehicle can travel in the center better.

[0100] refer to Figure 6 , Figure 6 FIG. 4 is a flow chart of a second embodiment of a vehicle centering driving method according to the present invention.

[0101] Based on the first embodiment, the vehicle centering driving method of this embodiment includes, in step S30:

[0102] Step S31: acquiring imaging internal parameters of an image acquisition unit, wherein the image acquisition unit is used to acquire the road surface image.

[0103] It is understandable that the intrinsic parameters of the image acquisition device, also known as the internal attribute parameters of the image acquisition device, mainly include focal length, principal point (optical center) coordinates, distortion coefficients, etc. These parameters are used to describe the imaging process of the image acquisition device. The intrinsic parameters are usually determined when the image acquisition device is calibrated, and are fixed for a specific image acquisition device model and do not change over time. Once the intrinsic parameters of the image acquisition device are determined, they usually remain unchanged during the use of the image acquisition device. The extrinsic parameters of the image acquisition device, also known as the external parameters of the image acquisition device, describe the position and posture of the image acquisition device in the world coordinate system. Usually include rotation matrices and translation vectors, which are used to calculate the position and posture of objects in the three-dimensional space in the image captured by the image acquisition device. The value of the extrinsic parameter can be obtained by image acquisition device posture estimation or three-dimensional reconstruction. In many computer vision tasks, such as SLAM (simultaneous localization and mapping), three-dimensional reconstruction, virtual reality, etc., the external parameters of the image acquisition device are very important parameters.

[0104] Step S32: converting the pixel coordinates into an imaging coordinate system according to the imaging intrinsic reference to obtain a reference coordinate.

[0105] It is understandable that the imaging coordinate system can be the coordinate system where the image acquisition device is located. The imaging coordinate system is a three-dimensional rectangular coordinate system established with the focal center of the camera as the origin and the optical axis as the Z axis. In this coordinate system, the x-axis and the y-axis are parallel to the X and Y axes of the image, while the z-axis is the optical axis of the camera, which is perpendicular to the graphic plane. This coordinate system is established to better understand and describe the process of light being projected onto the image plane through the lens. The image coordinate system is a two-dimensional coordinate system established on the image plane. Its origin is located at the intersection of the Z axis and the image plane in the imaging coordinate system, that is, the intersection of the optical axis and the image plane. The X axis and the Y axis are parallel to the X axis and the Y axis of the imaging coordinate system, respectively. The establishment of this coordinate system helps us describe and locate feature points on the image plane.

[0106] It should be understood that since the midpoint of the planned route needs to be sent to the on-board controller to control the vehicle's driving, and the on-board controller is based on the vehicle body coordinate system, the midpoint of the route is in the pixel coordinate system, and the midpoint of the pixel coordinate system needs to be transformed and transferred to the vehicle body's coordinate system based on the camera's intrinsic and extrinsic parameters.

[0107] It should be noted that the step of converting the pixel coordinates to the imaging coordinate system according to the imaging intrinsic parameters to obtain the reference coordinates includes: acquiring the focal length and imaging depth of the image acquisition unit; determining the imaging coordinate system according to the imaging intrinsic parameters, and determining the pixel coordinate system according to the pixel coordinates; obtaining the projection relationship between the imaging coordinate system and the pixel coordinate system according to the focal length and the imaging depth; obtaining an imaging coordinate system conversion matrix according to the projection relationship and the imaging intrinsic parameters; and converting the pixel coordinates to the imaging coordinate system according to the imaging coordinate system conversion matrix to obtain the reference coordinates.

[0108] It is understandable that when the camera forms an image, it maps the objects in the three-dimensional world to the two-dimensional image plane. This process can be described by a geometric model. For details, please refer to Figure 7 , the conversion from pixel coordinate system to camera coordinate system also requires the concept of image coordinate system, which involves three coordinate systems: pixel coordinate system (u, v): describes the position of the image point on the photo after the object is imaged, the unit is (number of pixels). Image coordinate system (x, y): describes the projection transmission relationship of the object from the pixel coordinate system to the image coordinate system, the unit is m. Camera coordinate system (x c ,y c , z c ): A coordinate system established on the camera, defined to describe the position of an object from the camera's perspective. It serves as an intermediate link between the world coordinate system and the image / pixel coordinate system. The unit is m.

[0109] It should be understood that a photo is an image composed of pixels. The coordinates of the pixels are (u, v). They are not the coordinates on the physical imaging plane, but only the positions of the pixels on the image. Therefore, they cannot be directly used to restore the three-dimensional coordinates. For details, please refer to Figure 8 , Figure 8 Schematic diagram of the imaging coordinate system space. Although they are on the same imaging plane, the origin of the pixel coordinate system is in the upper left corner of the image. It describes the row and column of the pixel point, and the unit is the number of pixels. The origin of the image coordinate system is at the center of the image, which describes the actual distance of the object from the optical center, and the unit is m.

[0110] It should be noted that based on Figure 7 and Figure 8 It can be seen that the conversion relationship between the pixel coordinates (u, v) and the imaging coordinates (x, y) is as follows:

[0111]

[0112] Among them, u and v are the horizontal and vertical coordinates in the pixel coordinate system, x and y are the coordinates in the image coordinate system, (x0, y0) is the center of the imaging plane, that is, the coordinates of the origin of the image coordinate system in the pixel coordinate system; dx and dy represent the lengths of the physical pixel in the directions of the two coordinate axes respectively; x / dx represents how many pixels the point occupies in the x direction on the imaging plane, which can be specifically expressed in the following matrix form:

[0113]

[0114] Further inverse transformation of the matrix yields:

[0115]

[0116] refer to Figure 6 , the three-dimensional coordinates in the imaging coordinate system (x c ,y c , z c ) and the image acquisition device O intersect the imaging plane at a point (x, y). According to the proportional relationship, we get:

[0117]

[0118] Among them, f represents the focal length, zc is the depth, that is, the distance to the target point, which can be directly obtained by the depth camera. Through the above formula, we can get the projection process from the camera coordinate system to the image coordinate system:

[0119]

[0120] Among them, x c ,y c , z c is the coordinate in the camera coordinate system, and its inverse is:

[0121]

[0122] Among them, the internal parameter matrix K is a known quantity that has been determined when the image acquisition device leaves the factory:

[0123]

[0124] In summary, the transformation relationship from the pixel coordinate system to the imaging coordinate system is:

[0125]

[0126] Step S33: performing coordinate transformation on the reference coordinates according to the transformation matrix of the vehicle coordinate system to obtain the vehicle coordinates of the midpoint of the line.

[0127] It can be understood that the transformation matrix includes a rotation transformation matrix and a translation transformation matrix.

[0128] It should be understood that the coordinate transformation from the imaging coordinate system to the vehicle coordinate system belongs to a rigid body transformation, which can be achieved by relying on a rotation transformation and a translation transformation, and the rotation transformation and the translation transformation are calculated by a rotation transformation matrix and a translation transformation matrix.

[0129] It should be noted that the coordinate transformation of the reference coordinates according to the transformation matrix of the vehicle-mounted coordinate system to obtain the vehicle-mounted coordinates of the midpoint of the route includes: acquiring the intrinsic parameter matrix and the extrinsic parameter matrix of the image acquisition unit; obtaining the transformation matrix of the imaging coordinate system and the vehicle-mounted coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix, the transformation matrix including a rotation transformation matrix and a translation transformation matrix; performing coordinate system transformation on the reference coordinates according to the rotation transformation matrix and the translation transformation matrix to obtain the vehicle-mounted coordinates of the midpoint of the route.

[0130] It should be noted that the conversion from the imaging coordinate system to the vehicle coordinate system can be referred to Fig. 9 , based on reference Fig. 9 It can be seen that the transformation relationship is as follows:

[0131]

[0132] Among them, x w ,y w , z w is the coordinate in the vehicle coordinate system, R is the rotation transformation matrix, T is the translation transformation matrix, and the formula can be further converted into the following formula:

[0133]

[0134] In summary, the transformation formula from the imaging coordinate system to the vehicle coordinate system is as follows:

[0135]

[0136] Among them, K1 is the intrinsic parameter matrix of the camera, and K2 is the extrinsic parameter matrix of the camera, as follows:

[0137]

[0138] It is understandable that after the conversion is completed, the converted coordinates (xw, yw) are sent to the vehicle controller to control the vehicle to travel to the coordinates (xw, yw) in the vehicle coordinate system.

[0139] In this embodiment, an image acquisition device is used to acquire a road surface image according to a preview distance, and then the midpoint of the route is obtained according to the drivable area in the road surface image. The coordinates of the midpoint of the route in the image are converted to the vehicle coordinate system of the ego vehicle through the imaging coordinate system of the image acquisition device, thereby obtaining the target point where the ego vehicle is centered in the vehicle coordinate system. Based on this point, the steering angle of the ego vehicle is controlled to enable accurate centered driving.

[0140] Reference Fig. 9 , Fig. 9 It is a structural block diagram of the first embodiment of the vehicle centering driving device of the present invention.

[0141] like Fig. 9 As shown, the vehicle centering driving device proposed in the embodiment of the present invention includes:

[0142] An image processing module 10 is used to obtain a road image of the vehicle and identify a drivable area in the road image;

[0143] A coordinate conversion module 20 is used to perform image analysis on the drivable area to obtain the line midpoint of the drivable area and the pixel coordinates of the line midpoint;

[0144] The coordinate conversion module 20 is further used to perform coordinate conversion on the pixel coordinates based on the vehicle-mounted coordinate system of the vehicle to obtain the vehicle-mounted coordinates of the midpoint of the route;

[0145] The centering driving module 30 is used to obtain the steering angle of the vehicle according to the vehicle-mounted coordinates and the vehicle coordinates, and control the vehicle to travel to the middle point of the route according to the steering angle of the vehicle to achieve centering driving of the vehicle.

[0146] This embodiment acquires road surface images during driving and analyzes the road surface images to obtain the midpoint of the line in the image, converts the pixel coordinates of the midpoint of the line in the road surface image into coordinates in the vehicle coordinate system, obtains the steering angle of the vehicle when it travels to the midpoint of the line based on the coordinates in the vehicle coordinate system, controls the vehicle to travel in the center based on the steering angle, enables the vehicle to determine the target point instead of the lane line through the drivable area, can control the vehicle travel more stably, and can flexibly adjust the preview distance according to the vehicle speed, so that the vehicle can travel in the center better.

[0147] In one embodiment, the coordinate conversion module 20 is further used to obtain the longitudinal resolution, the scale factor and the vehicle speed of the drivable area;

[0148] Obtaining a preview distance of the drivable area according to the longitudinal resolution, the proportional coefficient, and the vehicle speed;

[0149] Segmenting the drivable area based on the preview distance to obtain segmented images;

[0150] The image midpoint of the segmented image is identified, and the line midpoint of the drivable area and the pixel coordinates of the line midpoint are obtained according to the image midpoint.

[0151] In one embodiment, the coordinate conversion module 20 is further used to obtain reference pixel values ​​of each pixel point of the segmented image;

[0152] Sorting the reference pixel points according to preset standard road surface pixel values ​​to obtain a reference pixel point set;

[0153] The midpoint of the line in the drivable area and the pixel coordinates of the midpoint of the line are obtained according to the pixel values ​​of the first pixel point and the last pixel point in the reference pixel point set.

[0154] In one embodiment, the coordinate conversion module 20 is further used to obtain imaging internal parameters of an image acquisition unit, and the image acquisition unit is used to obtain the road surface image;

[0155] Convert the pixel coordinates to an imaging coordinate system according to the imaging intrinsic reference to obtain a reference coordinate;

[0156] The reference coordinates are transformed according to the transformation matrix of the vehicle coordinate system to obtain the vehicle coordinates of the midpoint of the line.

[0157] In one embodiment, the coordinate conversion module 20 is further used to obtain the focal length and imaging depth of the image acquisition unit;

[0158] Determine an imaging coordinate system according to the imaging intrinsic parameters, and determine a pixel coordinate system according to the pixel coordinates;

[0159] Obtaining a projection relationship between the imaging coordinate system and the pixel coordinate system according to the focal length and the imaging depth;

[0160] Obtaining an imaging coordinate system conversion matrix according to the projection relationship and the imaging internal parameters;

[0161] The pixel coordinates are converted to an imaging coordinate system according to the imaging coordinate system conversion matrix to obtain reference coordinates.

[0162] In one embodiment, the coordinate conversion module 20 is further used to obtain an intrinsic parameter matrix and an extrinsic parameter matrix of the image acquisition unit;

[0163] Obtaining a transformation matrix of the imaging coordinate system and the vehicle coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix, wherein the transformation matrix includes a rotation transformation matrix and a translation transformation matrix;

[0164] The reference coordinates are transformed into a coordinate system according to the rotation transformation matrix and the translation transformation matrix to obtain the vehicle-mounted coordinates of the midpoint of the route.

[0165] In one embodiment, the centering driving module 30 is further used to obtain the vehicle wheelbase of the vehicle;

[0166] Obtaining a relative relationship between the steering curvature of the vehicle and the front wheel turning angle according to the geometric kinematics model of the vehicle;

[0167] Obtaining a steering angle of the vehicle based on the relative relationship, the vehicle wheelbase, the vehicle-mounted coordinates, and the vehicle coordinates;

[0168] The vehicle is controlled to travel to the midpoint of the route according to the steering angle of the vehicle, so as to realize the centering driving of the vehicle.

[0169] In addition, to achieve the above-mentioned purpose, the present invention also proposes a vehicle centering driving device, which includes: a memory, a processor, and a vehicle centering driving program stored in the memory and executable on the processor, and the vehicle centering driving program is configured to implement the steps of the vehicle centering driving method described above.

[0170] Since the vehicle centering driving device adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0171] In addition, an embodiment of the present invention further proposes a storage medium, on which a vehicle centering driving program is stored. When the vehicle centering driving program is executed by a processor, the steps of the vehicle centering driving method described above are implemented.

[0172] Since the storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought by the technical solutions of the above embodiments, which will not be described one by one here.

[0173] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0174] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0175] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the vehicle centering driving method provided in any embodiment of the present invention, and will not be repeated here.

[0176] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0177] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0178] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0180] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A vehicle centering driving method, characterized in that: The vehicle centering driving method comprises: Acquire a road image of the vehicle and identify a drivable area in the road image; Performing image analysis on the drivable area to obtain the midpoint of the line in the drivable area and the pixel coordinates of the midpoint of the line; Performing coordinate transformation on the pixel coordinates based on the vehicle-mounted coordinate system of the vehicle to obtain the vehicle-mounted coordinates of the midpoint of the line; Obtaining a steering angle of the vehicle according to the vehicle-mounted coordinates and the vehicle coordinates, and controlling the vehicle to travel to the midpoint of the route according to the steering angle of the vehicle to achieve vehicle centering; The performing image analysis on the drivable area to obtain the line midpoint of the drivable area and the pixel coordinates of the line midpoint includes: Obtaining the longitudinal resolution, scale factor and vehicle speed of the drivable area; Obtaining a preview distance of the drivable area according to the longitudinal resolution, the proportional coefficient, and the vehicle speed; Segmenting the drivable area based on the preview distance to obtain segmented images; The image midpoint of the segmented image is identified, and the line midpoint of the drivable area and the pixel coordinates of the line midpoint are obtained according to the image midpoint.

2. The vehicle centering driving method according to claim 1, characterized in that: The step of identifying the image midpoint of the segmented image and obtaining the line midpoint of the drivable area and the pixel coordinates of the line midpoint according to the image midpoint includes: Obtaining a reference pixel value of each pixel point of the segmented image; Sorting the reference pixel points according to preset standard road surface pixel values ​​to obtain a reference pixel point set; The midpoint of the line in the drivable area and the pixel coordinates of the midpoint of the line are obtained according to the pixel values ​​of the first pixel point and the last pixel point in the reference pixel point set.

3. The vehicle centering driving method according to claim 1, characterized in that: The step of performing coordinate conversion on the pixel coordinates based on the vehicle-mounted coordinate system of the vehicle to obtain the vehicle-mounted coordinates of the midpoint of the route includes: Acquiring an imaging internal parameter of an image acquisition unit, wherein the image acquisition unit is used to acquire the road surface image; Convert the pixel coordinates to an imaging coordinate system according to the imaging intrinsic reference to obtain a reference coordinate; The reference coordinates are transformed according to the transformation matrix of the vehicle coordinate system to obtain the vehicle coordinates of the midpoint of the line.

4. The vehicle centering driving method according to claim 3, characterized in that: The step of converting the pixel coordinates into an imaging coordinate system according to the imaging intrinsic reference to obtain a reference coordinate comprises: Obtaining the focal length and imaging depth of the image acquisition unit; Determine an imaging coordinate system according to the imaging intrinsic parameters, and determine a pixel coordinate system according to the pixel coordinates; Obtaining a projection relationship between the imaging coordinate system and the pixel coordinate system according to the focal length and the imaging depth; Obtaining an imaging coordinate system conversion matrix according to the projection relationship and the imaging internal parameters; The pixel coordinates are converted to an imaging coordinate system according to the imaging coordinate system conversion matrix to obtain reference coordinates.

5. The vehicle centering driving method according to claim 3, characterized in that: The step of performing coordinate transformation on the reference coordinate according to the transformation matrix of the vehicle-mounted coordinate system to obtain the vehicle-mounted coordinate of the midpoint of the line includes: Obtaining the intrinsic parameter matrix and extrinsic parameter matrix of the image acquisition unit; Obtaining a transformation matrix of the imaging coordinate system and the vehicle coordinate system according to the intrinsic parameter matrix and the extrinsic parameter matrix, wherein the transformation matrix includes a rotation transformation matrix and a translation transformation matrix; The reference coordinates are transformed into a coordinate system according to the rotation transformation matrix and the translation transformation matrix to obtain the vehicle-mounted coordinates of the midpoint of the route.

6. The vehicle centering driving method according to claim 1, characterized in that: The method of obtaining a steering angle of the vehicle according to the vehicle-mounted coordinates and the vehicle coordinates, and controlling the vehicle to travel to the middle point of the route according to the steering angle of the vehicle to realize vehicle centering driving includes: Obtaining the wheelbase of the vehicle; Obtaining a relative relationship between the steering curvature of the vehicle and the front wheel turning angle according to the geometric kinematics model of the vehicle; Obtaining a steering angle of the vehicle based on the relative relationship, the vehicle wheelbase, the vehicle-mounted coordinates, and the vehicle coordinates; The vehicle is controlled to travel to the midpoint of the route according to the steering angle of the vehicle, so as to realize the centering driving of the vehicle.

7. A vehicle centering driving device, characterized in that: The vehicle centering driving device comprises: An image processing module, used to obtain a road image of the vehicle and identify a drivable area in the road image; A coordinate conversion module, used for performing image analysis on the drivable area to obtain the line midpoint of the drivable area and the pixel coordinates of the line midpoint; The coordinate conversion module is also used for the coordinate conversion module, and is also used to perform coordinate conversion on the pixel coordinates based on the vehicle-mounted coordinate system of the vehicle to obtain the vehicle-mounted coordinates of the midpoint of the route; A centering driving module, used for obtaining a steering angle of the vehicle according to the vehicle-mounted coordinates and the vehicle coordinates, and controlling the vehicle to travel to the middle point of the route according to the steering angle of the vehicle to realize centering driving of the vehicle; The coordinate conversion module is further used to obtain the longitudinal resolution, the scale factor and the speed of the vehicle in the drivable area; obtain the preview distance of the drivable area according to the longitudinal resolution, the scale factor and the speed of the vehicle; divide the drivable area based on the preview distance to obtain a segmented image; identify the image midpoint of the segmented image, and obtain the line midpoint of the drivable area and the pixel coordinates of the line midpoint according to the image midpoint.

8. A vehicle centering driving device, characterized in that: The device comprises: a memory, a processor, and a vehicle centering driving program stored in the memory and executable on the processor, wherein the vehicle centering driving program is configured to implement the vehicle centering driving method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a vehicle centering driving program, and when the vehicle centering driving program is executed by the processor, the vehicle centering driving method according to any one of claims 1 to 6 is implemented.

Images