Vehicle auxiliary line calibration and display method and device, related equipment and program product

Abstract

The invention discloses a vehicle auxiliary line calibration and display method and device, related equipment and a program product. According to a vehicle kinematics model, the position of an instantaneous rotation center on a straight line where a vehicle rear wheel is located is calculated at different front wheel rotation angles, and for a target structural part needing to draw a vehicle auxiliary line, the position of the instantaneous rotation center is calculated; calculating the distance from the position of the target structural member to the instantaneous rotation center, drawing an arc by taking the instantaneous rotation center as a circle center and the distance as a radius, selecting a section with a set length on the arc as a driving trajectory of the target structural member, and calculating the driving trajectory of the target structural member according to the position of the driving trajectory of the target structural member in the world coordinate system. The calibration position of the driving trajectory of the target structural member in the image shot by the camera is determined according to the coordinate system of the camera and the conversion relation between the world coordinate system and the coordinate system of the image shot by the front camera and the rear camera, and then the vehicle auxiliary line display work can be completed according to the calibration position. The vehicle auxiliary line calibration method is higher in automation degree, and the customization demand of the vehicle auxiliary line style is supported when the vehicle auxiliary line is displayed.

Description

Technical Field

[0001] The present application relates to the field of vehicle safety technology, and more specifically, to a method, device, related equipment and computer program product for calibrating and displaying vehicle auxiliary lines. Background Art

[0002] In the field of active vehicle safety, the front and rear cameras of the vehicle can capture surrounding images, and the driving trajectory of the wheels and the outer edge of the vehicle's rearview mirror can be displayed in the images captured by the camera, also known as the auxiliary line. As the steering wheel drives the vehicle to turn, the auxiliary line also changes to remind the user of the vehicle's next driving trajectory and improve driving safety.

[0003] In order to achieve the purpose of dynamically changing the auxiliary line with the steering wheel, the auxiliary line needs to be calibrated in advance, that is, to determine the calibration position of the auxiliary line in the image taken by the camera at different vehicle turning angles. The traditional method generally uses a radar map with a grid laid on the front and rear of the vehicle, and then actually drives the car through the grid radar map with different steering wheel angles to obtain the driving track of the wheel on the grid radar map, and then complete the calibration work. This method relies on a lot of manual participation, such as manual driving of the car, manual drawing of track lines on the grid radar map through track drawing tools, etc., which takes up a lot of human resources and has a low degree of intelligence. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a method, device, related equipment and computer program product for calibrating and displaying vehicle auxiliary lines, so as to improve the intelligence level of the vehicle auxiliary line calibration process. The specific scheme is as follows:

[0005] In a first aspect, a vehicle auxiliary line calibration method is provided, comprising:

[0006] According to the vehicle kinematic model, the position of the instantaneous rotation center on the straight line where the rear wheels of the vehicle are located is calculated at different front wheel turning angles of the vehicle;

[0007] For each target structural part on the vehicle, calculating the distance from the position of the target structural part to the instantaneous rotation center, the target structural part being a designated vehicle structural part for which a vehicle auxiliary line needs to be drawn;

[0008] An arc is drawn with the instantaneous rotation center as the center and the distance as the radius, and an arc of a set length located on a side of a first straight line away from the vehicle body is taken as a driving trajectory of the target structural component, wherein the first straight line is a straight line spaced from the outer contour of the vehicle by a set safety distance value;

[0009] According to the position of the driving trajectory of the target structural part in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras, the calibrated position of the driving trajectory of the target structural part in the images taken by the front and rear cameras is determined.

[0010] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the target structural member includes each wheel of the vehicle;

[0011] The process of calculating the distance from the position of the target structural component to the instantaneous rotation center includes:

[0012] Calculate the distances from the outer wheel point position and the inner wheel point position of each wheel to the instantaneous rotation center;

[0013] The process of obtaining the driving trajectory of the target structural component includes:

[0014] In the forward driving scenario, for each front wheel of the vehicle: with the instantaneous rotation center as the center, draw an arc with the distances from the outer wheel point position and the inner wheel point position of the front wheel to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain the outer wheel driving trajectory line and the inner wheel driving trajectory line of the front wheel, and the outer wheel driving trajectory line and the inner wheel driving trajectory line of the front wheel form the driving trajectory line of the front wheel;

[0015] The process of determining the calibrated position of the driving trajectory of the target structural component in the image captured by the front camera includes:

[0016] According to the position of the driving track line of the front wheel in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front camera, the calibrated position of the driving track line of the front wheel in the image taken by the front camera is determined.

[0017] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of obtaining the driving trajectory of the target structural component further includes:

[0018] In the reversing scenario, for each rear wheel of the vehicle: with the instantaneous rotation center as the center, draw an arc with the distances from the outer wheel point position and the inner wheel point position of the rear wheel to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain the outer wheel driving trajectory line and the inner wheel driving trajectory line of the rear wheel, and the outer wheel driving trajectory line and the inner wheel driving trajectory line of the rear wheel form the driving trajectory line of the rear wheel;

[0019] The process of determining the calibrated position of the driving trajectory of the target structural component in the image captured by the rear camera includes:

[0020] According to the position of the driving track line of the rear wheel in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the rear camera, the calibrated position of the driving track line of the rear wheel in the image taken by the rear camera is determined.

[0021] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the target structural member includes an outer edge of a vehicle rearview mirror;

[0022] The process of calculating the distance from the position of the target structural component to the instantaneous rotation center includes:

[0023] Calculating the distance from the position of the outer edge of the vehicle rearview mirror to the instantaneous rotation center;

[0024] The process of obtaining the driving trajectory of the target structural component includes:

[0025] Draw an arc with the instantaneous rotation center as the center and the distance from the position of the outer edge of the vehicle rearview mirror to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain a driving trajectory line of the outer edge of the vehicle rearview mirror;

[0026] The process of determining the calibrated position of the driving trajectory of the target structural component in the images taken by the front and rear cameras includes:

[0027] According to the position of the driving trajectory line of the outer edge of the vehicle rearview mirror in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras, the calibration position of the driving trajectory line of the outer edge of the vehicle rearview mirror in the images taken by the front and rear cameras is determined.

[0028] In one possible design, in another implementation of the first aspect of the embodiment of the present application, in a forward driving scenario, the first straight line is located in front of the vehicle, parallel to the front bumper and spaced apart by a set safety distance value; in a reversing scenario, the first straight line is located behind the vehicle, parallel to the rear bumper and spaced apart by a set safety distance value.

[0029] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the transformation relationship between the world coordinate system and the image coordinate system captured by the front and rear cameras is calculated based on the extrinsic rotation matrix, extrinsic translation vector and intrinsic matrix of the front and rear cameras.

[0030] In a second aspect, a method for displaying a vehicle auxiliary line is provided, comprising:

[0031] In response to a vehicle auxiliary line display instruction under a target scene, an image captured by a target camera is acquired, wherein when the target scene is a forward driving scene, the target camera is a front camera of the vehicle, and when the target scene is a reverse driving scene, the target camera is a rear camera of the vehicle;

[0032] Get the configured vehicle auxiliary line UI image and the current front wheel steering angle of the vehicle;

[0033] In the configured vehicle auxiliary line calibration information file, searching for the target calibration position of the vehicle auxiliary line in the image captured by the target camera at the current front wheel turning angle, the correspondence between the different front wheel turning angles recorded in the vehicle auxiliary line calibration information file and the calibration positions of the vehicle auxiliary line in the images captured by the front and rear cameras is obtained by using the vehicle auxiliary line calibration method of any one of the first aspects of the present application;

[0034] The vehicle auxiliary line UI picture is rendered into the image captured by the target camera according to the target calibration position, and the rendered image is output and displayed.

[0035] In a possible design, in another implementation of the second aspect of the embodiment of the present application, the process of rendering the auxiliary line UI picture into the image captured by the target camera according to the target calibration position includes:

[0036] The vehicle auxiliary line UI picture is rendered to the target calibration position in the image taken by the target camera through the configured rendering engine.

[0037] In a possible design, in another implementation of the second aspect of the embodiment of the present application, the process of rendering the auxiliary line UI picture to the target calibration position in the image captured by the target camera through the configured rendering engine includes:

[0038] Divide the texture coordinates of the vehicle auxiliary line UI image and the target calibration position into n parts with equal distances, and correspond them one by one to n groups of texture coordinates and vertex coordinates of the texture map;

[0039] Through the configured rendering engine, the vehicle auxiliary line UI picture is rendered into the image taken by the camera according to the correspondence between each set of texture coordinates and the vertex coordinates of the texture map.

[0040] In a possible design, in another implementation of the second aspect of the embodiment of the present application, before outputting the rendered image for display, the method further includes:

[0041] The rendered image is cropped and transformed, and the processed image is output and displayed.

[0042] In a third aspect, a vehicle auxiliary line calibration device is provided, comprising:

[0043] An instantaneous rotation center determination unit is used to calculate the position of the instantaneous rotation center on the straight line where the rear wheels of the vehicle are located when the front wheels of the vehicle have different turning angles according to the vehicle kinematic model;

[0044] A distance calculation unit, used for calculating the distance from the position of each target structural part on the vehicle to the instantaneous rotation center, wherein the target structural part is a designated vehicle structural part for which a vehicle auxiliary line needs to be drawn;

[0045] an arc drawing unit, used for drawing an arc with the instantaneous rotation center as the center and the distance as the radius, taking an arc of a set length on a side of a first straight line away from the vehicle body as a driving trajectory of the target structural component, wherein the first straight line is a straight line spaced from the outer contour of the vehicle by a set safety distance value;

[0046] A coordinate conversion unit is used to determine the calibrated position of the driving trajectory line of the target structural part in the images taken by the front and rear cameras according to the position of the driving trajectory line of the target structural part in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras.

[0047] In a fourth aspect, a vehicle auxiliary line display device is provided, comprising:

[0048] An image acquisition unit, configured to acquire an image captured by a target camera in response to a vehicle auxiliary line display instruction in a target scene, wherein when the target scene is a forward driving scene, the target camera is a front camera of the vehicle, and when the target scene is a reverse driving scene, the target camera is a rear camera of the vehicle;

[0049] A car auxiliary line UI picture acquisition unit is used to acquire the configured car auxiliary line UI picture;

[0050] A turning angle acquisition unit, used to acquire the current front wheel turning angle of the vehicle;

[0051] a calibration position search unit, used to search, in a configured vehicle auxiliary line calibration information file, a target calibration position of the vehicle auxiliary line in the image captured by the target camera at the current front wheel turning angle, wherein the correspondence between different front wheel turning angles recorded in the vehicle auxiliary line calibration information file and the calibration positions of the vehicle auxiliary line in the images captured by the front and rear cameras is obtained by using the vehicle auxiliary line calibration method of any one of the aforementioned first aspects of the present application;

[0052] An image rendering unit is used to render the vehicle auxiliary line UI picture into the image taken by the target camera according to the target calibration position, and output the rendered image for display.

[0053] In a fifth aspect, an electronic device is provided, comprising: a memory and a processor;

[0054] The memory is used to store programs;

[0055] The processor is used to execute the program to implement the various steps of the vehicle auxiliary line calibration method as described in any one of the first aspects of the present application, or to implement the various steps of the vehicle auxiliary line display method as described in any one of the second aspects of the present application.

[0056] In a sixth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the vehicle auxiliary line calibration method as described in any one of the first aspects of the present application are implemented, or the various steps of the vehicle auxiliary line display method as described in any one of the second aspects of the present application are implemented.

[0057] In the seventh aspect, a computer program product is provided, comprising a computer program. When the computer program is executed by a processor, the computer program implements the various steps of the vehicle auxiliary line calibration method as described in any one of the first aspects of the present application, or implements the various steps of the vehicle auxiliary line display method as described in any one of the second aspects of the present application.

[0058] By means of the above technical scheme, the present application can pre-calculate the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras according to the external parameters and internal parameters of the front and rear cameras on the vehicle, and then any position point in the world coordinate system can be converted into the image taken by the camera. On this basis, the positions of various components on the vehicle in the world coordinate system can be easily obtained. When calibrating the auxiliary line of the vehicle, according to the vehicle kinematic model, the position of the instantaneous rotation center on the straight line where the rear wheels of the vehicle are located is calculated at different front wheel turning angles of the vehicle. For each target structural part (such as wheels, outer edges of rearview mirrors, etc.) on the vehicle where the auxiliary line of the vehicle needs to be drawn, the distance from its position to the instantaneous rotation center is calculated, and an arc is drawn with the instantaneous rotation center as the center and the calculated distance as the radius, and the arc of the set length located on the first straight line away from the vehicle body is taken as the driving trajectory of the target structural part, that is, the position of the driving trajectory of the target structural part in the world coordinate system is obtained. According to the conversion relationship between the above-mentioned world coordinate system and the image coordinate system taken by the front and rear cameras, the calibrated position of the target structural part's driving trajectory in the images taken by the front and rear cameras can be obtained to complete the calibration of the vehicle auxiliary line. Obviously, this application does not require manual laying of grid radar maps, nor does it require users to actually drive the vehicle. It only needs to measure the position information of several components on the vehicle in the world coordinate system, and then the calibration of the vehicle auxiliary line can be completed automatically, saving labor costs, with a higher degree of automation and more accurate calibration results. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0060] Figure 1a-Figure 1c Several different reversing image display effects are illustrated respectively;

[0061] Figure 2 A schematic diagram of a flow chart of a vehicle auxiliary line calibration method provided in an embodiment of the present application;

[0062] Figure 3 The schematic diagram of the process of determining the left front wheel driving trajectory in the forward driving scenario is illustrated;

[0063] Figure 4 The schematic diagram of the process of determining the driving trajectory of the outer edge of the left rearview mirror in the forward driving scenario is illustrated;

[0064] Figure 5 A schematic diagram of a method for displaying auxiliary vehicle lines provided in an embodiment of the present application;

[0065] Figure 6a An example of a vehicle's rearview mirror outer edge driving track UI image effect;

[0066] Figure 6b An example of a wheel track UI image effect is shown;

[0067] Figure 7 A schematic diagram of the structure of a vehicle auxiliary line calibration device provided in an embodiment of the present application;

[0068] Figure 8 A schematic diagram of the structure of a vehicle auxiliary line display device provided in an embodiment of the present application;

[0069] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] Current vehicles are generally equipped with cameras that can capture images of the vehicle's surroundings, making it easier for users to understand the vehicle's surroundings while in the vehicle. In addition, in some modes, the vehicle auxiliary lines can be further displayed in the images captured by the camera, such as displaying the outer edges of the wheel mirrors and the driving trajectory of the wheels in the front and rear single views. In particular, the outer edges of the vehicle's rearview mirrors and the driving trajectory of the wheels displayed in the rear single view captured by the rear camera are called rear view vehicle auxiliary lines, which are the main components of the reversing imaging system.

[0072] Generally, in order to ensure that the camera can capture environmental images at a larger angle, a fisheye camera can be used for image acquisition.

[0073] Reference Figure 1a-Figure 1c :

[0074] Figure 1a An example of a display effect of displaying a vehicle auxiliary line in a rear view when the vehicle steering wheel angle is 0 degrees. Figure 1b This example illustrates an effect of displaying a vehicle auxiliary line in the rear view when the vehicle steering wheel turns 100 degrees to the left. Figure 1c This example illustrates an effect of displaying a vehicle auxiliary line in the rear view when the vehicle steering wheel turns 100 degrees to the right.

[0075] It should be noted that Figure 1a-Figure 1c The display effect shown is the effect after the original image taken by the fisheye camera is cropped and transformed. The original image taken by the fisheye camera may contain a vehicle bumper and objects at a longer distance. These images are not of concern to the user. Therefore, through cropping and transformation, only the image of the area of ​​concern to the user is retained.

[0076] Figure 1a-Figure 1c The auxiliary lines shown include the driving track lines of the outer edges of the left and right rearview mirrors and the driving track lines of the wheels on both sides. In addition, it also includes a warning line, which can be a straight line parallel to the vehicle bumper and separated by a certain distance (for example, 20-30cm), used to warn the distance between vehicles. Since the image taken by the fisheye camera has barrel distortion, the warning line in the figure looks like a curve.

[0077] The traditional method of calibrating the auxiliary lines of a vehicle generally relies on laying out a grid radar map at the front and rear of the vehicle, and then actually driving the car through the grid radar map at different steering wheel angles to obtain the driving tracks of the wheels and the outer edges of the rearview mirrors on the grid radar map to calibrate the track position of the auxiliary lines in the single view taken by the front and rear cameras. This method requires a high degree of manual participation, such as manual driving of the car, manual drawing of track lines on the grid radar map using track line drawing tools, etc., and the degree of intelligence is not high. In addition, the auxiliary line style only supports simple lines, and cannot flexibly respond to the customized needs of different users for the auxiliary line UI style.

[0078] The present application provides a method for calibrating a vehicle auxiliary line and a method for displaying a vehicle auxiliary line. The method for calibrating a vehicle auxiliary line can be applied to a terminal with data processing capabilities, such as a computer, a server, etc. The method for displaying a vehicle auxiliary line can be applied to a vehicle-mounted terminal to realize the function of outputting and displaying a vehicle auxiliary line image in a vehicle driving scene.

[0079] Next, combine Figure 2 The vehicle auxiliary line calibration method of the present application may include the following steps:

[0080] Step S100: Calculate the position of the instantaneous rotation center of the rear wheels of the vehicle on the straight line at different front wheel turning angles of the vehicle according to the vehicle kinematic model.

[0081] In the present application, a vehicle of the front wheel steering control type is taken as an example, that is, when the steering wheel is turned, only the angle of the front wheels of the vehicle is changed.

[0082] There is a fixed relationship between the steering wheel rotation angle and the front wheel rotation angle, and the front wheel rotation angle can be detected by a front wheel rotation angle sensor. The steering wheel rotation angle can also be detected by a steering wheel rotation sensor, and then converted and calculated to obtain the front wheel rotation angle.

[0083] In this step, each angle value within the rotatable angle range of the front wheel may be traversed. For example, within the rotatable angle range of the front wheel, an angle value θ is taken at each set degree interval (for example, at intervals of 0.1 degrees or other values).

[0084] Based on the vehicle kinematic model, the position of the instantaneous rotation center of the vehicle's rear wheels on the straight line when the vehicle's front wheels are at different angles θ can be calculated.

[0085] The process of calculating the position of the instantaneous rotation center in this step is performed in the world coordinate system. Based on the vehicle kinematic model and the positions of various structural parts on the vehicle, the position of the instantaneous rotation center can be calculated.

[0086] Step S110: For each target structural part on the vehicle, the distance from the position of the target structural part to the instantaneous rotation center is calculated, and the target structural part is a designated vehicle structural part for which a vehicle auxiliary line needs to be drawn.

[0087] Specifically, according to the needs of specific business, the vehicle structure that needs to draw the vehicle auxiliary line can be used as the target structure. For example, when it is required to display the wheel's driving track, the target structure can include the wheel. When it is required to display the driving track of the outer edge of the vehicle's rearview mirror, the target structure can include the outer edge of the vehicle's rearview mirror. Figure 1a As shown, the vehicle auxiliary line in the rear view of the example includes both the driving trajectory of the vehicle and the driving trajectory of the outer edge of the rearview mirror.

[0088] In this step, for each target structural part, its position in the world coordinate system can be measured in advance or obtained by consulting the vehicle manual, etc. After the position of the instantaneous rotation center is calculated in the previous step, the distance from the position of the target structural part to the instantaneous rotation center can be further calculated.

[0089] Step S120, drawing an arc with the instantaneous rotation center as the center and the distance as the radius, and taking an arc of a set length on the side of the first straight line away from the vehicle body as the driving trajectory of the target structural component.

[0090] The first straight line is a straight line with a set safety distance value from the outer contour of the vehicle. The first straight line can be understood as the starting point of the vehicle auxiliary line, and the set safety distance value can be about 20-30cm, which plays a warning role.

[0091] In a possible implementation, in a forward driving scenario, the first straight line may be located in front of the vehicle, parallel to the front bumper and spaced apart by a set safety distance value. In a reverse driving scenario, the first straight line may be located behind the vehicle, parallel to the rear bumper and spaced apart by a set safety distance value.

[0092] According to the vehicle kinematics model, the arc drawn with the instantaneous rotation center as the center and the distance from the target structure to the instantaneous rotation center as the radius is the trajectory that the target structure will travel in the future under the current vehicle front wheel turning state. In this step, the intersection of the drawn arc and the first straight line is used as the starting point of the driving trajectory line, and an arc of a set length is intercepted from the starting point along the arc to the side away from the vehicle body as the driving trajectory line of the target structure.

[0093] Generally, the above set length can be 3m or other values, and can be flexibly set according to business needs.

[0094] In this step, after the driving trajectory of the target structural part is obtained, the position of the driving trajectory of the target structural part in the world coordinate system can be determined.

[0095] Step S130: Determine the calibrated position of the target structural component's driving trajectory in the images taken by the front and rear cameras according to the position of the target structural component's driving trajectory in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras.

[0096] Specifically, the present application can calculate the conversion relationship between the world coordinate system and the image coordinate system captured by the front and rear cameras based on the extrinsic rotation matrix R, extrinsic translation vector T and intrinsic matrix K of the front and rear cameras.

[0097] In one possible implementation, the world coordinate system P is defined as world (X w ,Yw ,Z w ), where the x-axis is the front of the vehicle, the y-axis is the left side of the vehicle, and the z-axis is the top of the vehicle. The origin of the world coordinates is O world At ground level.

[0098] Define the camera coordinate system P camera (X c ,Y c ,Z c ), where the x-axis is to the right of the camera, the y-axis is below the camera, and the z-axis is in front of the camera.

[0099] Therefore, the world coordinate system P world (X w ,Y w ,Z w ) and the camera coordinate system P camera (X c ,Y c ,Z c ) has a rotation R and translation T relationship. Through the camera's external parameter rotation matrix R, external parameter translation vector T and internal parameter matrix K, we can find any point P in the world coordinate system. world (X w ,Y w ,Z w ) in the pixel coordinates P in the image captured by the camera image (u,v). The formula is as follows:

[0100]

[0101] That is, Z c P image =K(RP word +t)=KTP word =KP camera .

[0102] In the world coordinate system P world (X w ,Y w ,Z w ) can easily measure the world coordinate system positions of the vehicle's front axle, rear axle, front bumper, rear bumper, inner wheel points, outer wheel points, and outer edges of the left and right rearview mirrors.

[0103] Of course, the above is only an example of an optional definition method of the world coordinate system and the camera coordinate system. In addition, the world coordinate system and the camera coordinate system can be defined separately in other ways, and the conversion relationship between the two coordinate systems can be calculated accordingly.

[0104] In step S130, the calibrated position of the target structural component's driving trajectory in the images taken by the front and rear cameras can be determined according to the position of the target structural component's driving trajectory in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras.

[0105] The vehicle auxiliary line calibration method provided in the embodiment of the present application does not require manual laying of grid radar maps, nor does it require users to actually drive the vehicle. It only needs to measure the position information of several components on the vehicle in the world coordinate system, and then the calibration of the vehicle auxiliary line can be completed automatically, saving labor costs, and has a higher degree of automation and more accurate calibration results.

[0106] According to the needs of the business scenario, the target structural parts that need to draw the vehicle auxiliary line may include wheels, outer edges of rearview mirrors, etc. Next, the calibration process of the driving track of wheels and outer edges of rearview mirrors is introduced respectively.

[0107] 1. For each wheel of the vehicle

[0108] Calculate the distance from the outer wheel point position and the inner wheel point position of the wheel to the instantaneous rotation center.

[0109] In the forward driving scenario, for each front wheel of the vehicle: with the instantaneous rotation center as the center, draw an arc with the distance from the outer wheel point position and the inner wheel point position of the front wheel to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain the outer wheel driving trajectory line and the inner wheel driving trajectory line of the front wheel, and the driving trajectory line of the front wheel is composed of the outer wheel driving trajectory line and the inner wheel driving trajectory line of the front wheel.

[0110] According to the position of the driving track line of the front wheel in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front camera, the calibration position of the driving track line of the front wheel in the image taken by the front camera is determined.

[0111] Reference Figure 3 Taking the left front wheel as an example, the process of determining the driving trajectory of the left front wheel in a forward driving scenario is introduced.

[0112] In the world coordinate system P world (X w ,Y w ,Z w ) can be used to easily measure the vehicle's wheelbase wheelBase.

[0113] wheelBase=frontwheelBase+backwheelBase

[0114] Among them, frontwheelBase represents the distance from the front axle to the Y axis, and backwheelBase represents the distance from the rear axle to the Y axis.

[0115] According to the vehicle kinematics model, the front wheel turning angle is taken as θ, and the position of the instantaneous rotation center P0 on the straight line where the rear axle of the vehicle is located can be obtained through trigonometric function knowledge:

[0116]

[0117] The outer wheel point position P1 and the inner wheel point position P2 of the left front wheel of the vehicle can be conveniently measured in the world coordinate system.

[0118] An arc is drawn with the instantaneous rotation center P0 as the center and the distance from P1 to P0 as the radius, and an arc is drawn with the distance from P2 to P0 as the radius.

[0119] The intersection of the arc drawn above and the warning line (first straight line) in front of the vehicle is the starting point of the vehicle auxiliary line. The trajectory of the arc is regarded as the movement trajectory of the inner and outer wheels of the vehicle. Generally, the length of the trajectory line can be set to a fixed length of 3m. That is, if Figure 3 The middle arc line L1 is the outer wheel trajectory line of the left front wheel, and the arc line L2 is the inner wheel trajectory line of the left front wheel.

[0120] The area between the arc L1 and the arc L2 is regarded as the area passed by the left front wheel as a whole.

[0121] Figure 3 In the example, only the driving trajectory of the left front wheel in the forward driving scenario is used for explanation. The principle of determining the driving trajectory of the right front wheel is the same as that of the left front wheel, and will not be described here.

[0122] Furthermore, in the reversing scenario, for each rear wheel of the vehicle: with the instantaneous rotation center as the center, draw an arc with the distance from the outer wheel point position and the inner wheel point position of the rear wheel to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain the outer wheel driving trajectory line and the inner wheel driving trajectory line of the rear wheel, and the outer wheel driving trajectory line and the inner wheel driving trajectory line of the rear wheel constitute the driving trajectory line of the rear wheel.

[0123] According to the position of the rear wheel's driving trajectory in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the rear camera, the calibration position of the rear wheel's driving trajectory in the image taken by the rear camera is determined.

[0124] 2. For each rearview mirror of the vehicle

[0125] The distance from the position of the outer edge of the vehicle rearview mirror to the instantaneous rotation center is calculated.

[0126] An arc is drawn with the instantaneous rotation center as the center and the distance from the outer edge of the vehicle rearview mirror to the instantaneous rotation center as the radius, and an arc of a set length located on the side of the first straight line away from the vehicle body is taken to obtain the driving trajectory line of the outer edge of the vehicle rearview mirror.

[0127] According to the position of the driving trajectory line of the outer edge of the vehicle rearview mirror in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras, the calibration position of the driving trajectory line of the outer edge of the vehicle rearview mirror in the images taken by the front and rear cameras is determined.

[0128] Reference Figure 4 Taking the outer edge of the left rearview mirror of the wheel as an example, it introduces the process of determining the driving trajectory line of the outer edge of the left rearview mirror in the forward driving scenario.

[0129] In the world coordinate system P world (X w ,Y w ,Z w ) can be used to easily measure the vehicle's wheelbase wheelBase.

[0130] wheelBase=frontwheelBase+backwheelBase

[0131] Among them, frontwheelBase represents the distance from the front axle to the Y axis, and backwheelBase represents the distance from the rear axle to the Y axis.

[0132] According to the vehicle kinematics model, the front wheel turning angle is taken as θ, and the position of the instantaneous rotation center P0 on the straight line where the rear axle of the vehicle is located can be obtained through trigonometric function knowledge:

[0133]

[0134] The outer edge position P3 of the left rearview mirror of the vehicle can be conveniently measured in the world coordinate system.

[0135] Draw an arc with the instantaneous rotation center P0 as the center and the distance from P3 to P0 as the radius.

[0136] The intersection of the arc drawn above and the warning line (first straight line) in front of the vehicle is the starting point of the vehicle auxiliary line. The trajectory of the arc is regarded as the motion trajectory of the outer edge of the left rearview mirror. Generally, the length of the trajectory line can be set to a fixed length of 3m. That is, if Figure 3 The middle arc line L3 is the driving trajectory line of the outer edge of the left rearview mirror.

[0137] Figure 4In the description, only the driving trajectory of the outer edge of the left rearview mirror in the forward driving scenario is used as an example. The principle of determining the driving trajectory of the outer edge of the right rearview mirror is the same as that of the outer edge of the left rearview mirror, and will not be described here. In addition, the process of determining the driving trajectory of the outer edge of the rearview mirror in the reverse scenario can also refer to the forward driving scenario, and will not be described here.

[0138] In some embodiments of the present application, a method for displaying a vehicle auxiliary line is further provided. The method for displaying a vehicle auxiliary line can be applied to a vehicle-mounted terminal, such as a vehicle computer terminal. The vehicle computer terminal can display an environment image captured by a camera and display the vehicle auxiliary line in the environment image.

[0139] Reference Figure 5 As shown, the method for displaying auxiliary lines of the present embodiment may include the following steps:

[0140] Step S200: In response to a vehicle auxiliary line display instruction in a target scene, an image captured by a target camera is acquired.

[0141] Specifically, the target scene may include a forward driving scene and a reverse driving scene, wherein the forward driving scene is a scene in which the vehicle is about to drive forward, and the reverse driving scene is a scene in which the vehicle is about to reverse backward. Depending on the target scene, images captured by different on-board cameras need to be displayed. For example, in a forward driving scene, images captured by the on-board front camera need to be displayed, so the target camera is the front camera; in a reverse driving scene, images captured by the on-board rear camera need to be displayed, so the target camera is the rear camera.

[0142] The front camera refers to a camera that can capture images of the front side of the vehicle, and is generally located at the front end of the vehicle. The rear camera refers to a camera that can capture images of the rear of the vehicle, and is generally located at the rear end of the vehicle.

[0143] The vehicle auxiliary line display instruction may be specifically triggered when a set condition is met. For example, when the gear is engaged in reverse gear, the vehicle auxiliary line display instruction is triggered. At this time, the image captured by the rear camera needs to be displayed, and the vehicle auxiliary line is displayed on the image. For another example, when it is detected that the road ahead is narrowing, the vehicle auxiliary line display instruction may be triggered. At this time, the image captured by the front camera needs to be displayed, and the vehicle auxiliary line is displayed on the image.

[0144] Step S210, obtaining the configured vehicle auxiliary line UI picture, and obtaining the current front wheel turning angle of the vehicle.

[0145] Specifically, the UI image of the auxiliary line can be provided in advance, and the present application solution supports users to customize different styles of auxiliary lines. Figure 6a and Figure 6b , respectively illustrating a driving trajectory UI diagram of the outer edge of a rearview mirror and a wheel driving trajectory UI diagram.

[0146] The auxiliary line styles supported by this application are not limited to simple lines, and can be flexibly customized according to user needs. Figure 6a As shown, the driving track of the outer edge of the rearview mirror can use a distance-indicating line divided into 7 sections, and the track drawn on the ground requires that the distance of each section decreases arbitrarily.

[0147] Furthermore, in this embodiment, the current front wheel angle of the vehicle is also obtained. The current front wheel angle of the vehicle can be obtained by converting the current steering wheel angle, or can be directly detected by a front wheel angle sensor.

[0148] Step S220, in the configured vehicle auxiliary line calibration information file, search for the target calibration position of the vehicle auxiliary line in the image taken by the target camera under the current front wheel turning angle, wherein the vehicle auxiliary line calibration information file records the correspondence between different front wheel turning angles and the calibration positions of the vehicle auxiliary line in the images taken by the front and rear cameras.

[0149] Specifically, according to the vehicle auxiliary line calibration method introduced in the aforementioned embodiment, the correspondence between different front wheel turning angles and the calibration positions of the vehicle auxiliary lines in the images taken by the front and rear cameras can be obtained in advance, and the correspondence can be further stored as a vehicle auxiliary line calibration information file, configured locally in the vehicle-mounted terminal, or in a server that communicates with the vehicle-mounted terminal.

[0150] It is understandable that for different models of vehicles, considering their different structures and sizes, corresponding vehicle auxiliary line calibration information files can be obtained respectively.

[0151] In this step, when the current front wheel turning angle of the vehicle is obtained, the target calibration position of the vehicle auxiliary line in the image taken by the target camera can be obtained by searching the vehicle auxiliary line calibration information file.

[0152] Step S230: render the vehicle auxiliary line UI picture into the image captured by the target camera according to the target calibration position, and output the rendered image for display.

[0153] Specifically, the target calibration position of the auxiliary line in the image has been determined in the aforementioned steps, so the auxiliary line UI picture can be rendered into the image according to the target calibration position, thereby obtaining a rendered image and outputting it for display.

[0154] In one possible implementation, considering that the image captured by the camera may contain areas that are not of interest to the user (such as a vehicle bumper, images at a distant location, etc.), the rendered image can be further cropped and transformed before being output and displayed, and the processed image can be output and displayed to retain the areas that the user is more concerned about.

[0155] By using the auxiliary line display method provided in this embodiment, the auxiliary line calibration information file can be generated in advance through the auxiliary line calibration method of the aforementioned embodiment, and then when the auxiliary line needs to be displayed, based on the current turning angle of the front wheel of the vehicle, the auxiliary line calibration information file can be searched to quickly obtain the target calibration position of the auxiliary line in the image, and then the configured auxiliary line UI picture can be rendered into the image taken by the camera in the form of a texture. The present application method supports users to customize various styles of auxiliary lines.

[0156] In the above step S230, the process of rendering the vehicle auxiliary line UI picture into the image captured by the target camera according to the target calibration position can be specifically implemented by a configured rendering engine.

[0157] For example, a rendering engine such as Opengl can be used to render the vehicle auxiliary line UI picture into the image captured by the target camera according to the target calibration position.

[0158] In some possible implementations, this embodiment provides a segmentation rendering method, specifically:

[0159] The texture coordinates of the UI image of the auxiliary line and the target calibration position are divided into n parts with equal distances, and correspond one to one to n groups of texture coordinates and vertex coordinates of the texture map. Among them, n can be set by the user. The larger the value of n, the smaller the granularity of the segmentation of the texture coordinates and the target calibration position, and the higher the accuracy after the final rendering. Of course, it may also increase the amount of calculation of the rendering engine. In this embodiment, the value of n can be 256, which can better balance the rendering effect and the amount of calculation.

[0160] Through the configured rendering engine, the vehicle auxiliary line UI picture is rendered into the image taken by the camera according to the correspondence between each set of texture coordinates and the vertex coordinates of the texture map.

[0161] The following is a description of the vehicle auxiliary line calibration device provided in an embodiment of the present application. The vehicle auxiliary line calibration device described below and the vehicle auxiliary line calibration method described above can be referenced to each other.

[0162] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of a vehicle auxiliary line calibration device disclosed in an embodiment of the present application.

[0163] like Figure 7 As shown, the device may include:

[0164] The instantaneous rotation center determination unit 11 is used to calculate the position of the instantaneous rotation center on the straight line where the rear wheels of the vehicle are located at different front wheel turning angles of the vehicle according to the vehicle kinematic model;

[0165] A distance calculation unit 12 is used to calculate the distance from the position of each target structural part on the vehicle to the instantaneous rotation center, wherein the target structural part is a designated vehicle structural part for which a vehicle auxiliary line needs to be drawn;

[0166] an arc drawing unit 13, configured to draw an arc with the instantaneous rotation center as the center and the distance as the radius, and take an arc of a set length on a side of a first straight line away from the vehicle body as a driving trajectory of the target structural component, wherein the first straight line is a straight line spaced from the outer contour of the vehicle by a set safety distance value;

[0167] The coordinate conversion unit 14 is used to determine the calibrated position of the driving trajectory line of the target structural component in the images taken by the front and rear cameras according to the position of the driving trajectory line of the target structural component in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras.

[0168] In one possible implementation, in a forward driving scenario, the first straight line is located in front of the vehicle, parallel to the front bumper and separated by a set safety distance value; in a reversing scenario, the first straight line is located behind the vehicle, parallel to the rear bumper and separated by a set safety distance value.

[0169] In one possible implementation, the target structural component includes each wheel of the vehicle. In this case:

[0170] The process of the distance calculation unit calculating the distance from the position of the target structural component to the instantaneous rotation center includes:

[0171] Calculate the distances from the outer wheel point position and the inner wheel point position of each wheel to the instantaneous rotation center;

[0172] The process of obtaining the driving trajectory of the target structural component by the arc drawing unit includes:

[0173] In the forward driving scenario, for each front wheel of the vehicle: with the instantaneous rotation center as the center, draw an arc with the distances from the outer wheel point position and the inner wheel point position of the front wheel to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain the outer wheel driving trajectory line and the inner wheel driving trajectory line of the front wheel, and the outer wheel driving trajectory line and the inner wheel driving trajectory line of the front wheel form the driving trajectory line of the front wheel;

[0174] The process of determining the calibrated position of the driving trajectory of the target structural component in the image captured by the front camera by the coordinate conversion unit includes:

[0175] According to the position of the driving track line of the front wheel in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front camera, the calibrated position of the driving track line of the front wheel in the image taken by the front camera is determined.

[0176] Furthermore, the process of obtaining the driving trajectory line of the target structural component by the arc drawing unit further includes:

[0177] In the reversing scenario, for each rear wheel of the vehicle: with the instantaneous rotation center as the center, draw an arc with the distances from the outer wheel point position and the inner wheel point position of the rear wheel to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain the outer wheel driving trajectory line and the inner wheel driving trajectory line of the rear wheel, and the outer wheel driving trajectory line and the inner wheel driving trajectory line of the rear wheel form the driving trajectory line of the rear wheel;

[0178] The process of the coordinate conversion unit determining the calibrated position of the driving trajectory of the target structural component in the image taken by the rear camera includes:

[0179] According to the position of the driving track line of the rear wheel in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the rear camera, the calibrated position of the driving track line of the rear wheel in the image taken by the rear camera is determined.

[0180] In some other possible implementations, the target structural component includes an outer edge of a vehicle rearview mirror. In this case:

[0181] The process of the distance calculation unit calculating the distance from the position of the target structural component to the instantaneous rotation center includes:

[0182] Calculating the distance from the position of the outer edge of the vehicle rearview mirror to the instantaneous rotation center;

[0183] The process of obtaining the driving trajectory of the target structural component by the arc drawing unit includes:

[0184] Draw an arc with the instantaneous rotation center as the center and the distance from the position of the outer edge of the vehicle rearview mirror to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain a driving trajectory line of the outer edge of the vehicle rearview mirror;

[0185] The process of determining the calibrated position of the driving trajectory of the target structural component in the images taken by the front and rear cameras by the coordinate conversion unit includes:

[0186] According to the position of the driving trajectory line of the outer edge of the vehicle rearview mirror in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras, the calibration position of the driving trajectory line of the outer edge of the vehicle rearview mirror in the images taken by the front and rear cameras is determined.

[0187] The device of the present application may also include:

[0188] The coordinate system conversion relationship calculation unit is used to calculate the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras according to the external parameter rotation matrix, external parameter translation vector and internal parameter matrix of the front and rear cameras.

[0189] The following is a description of a vehicle auxiliary line display device provided in an embodiment of the present application. The vehicle auxiliary line display device described below and the vehicle auxiliary line display method described above can be referenced to each other.

[0190] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a vehicle auxiliary line display device disclosed in an embodiment of the present application.

[0191] like Figure 8 As shown, the device may include:

[0192] An image acquisition unit 21 is used to acquire an image captured by a target camera in response to a vehicle auxiliary line display instruction in a target scene. When the target scene is a forward driving scene, the target camera is a front camera of the vehicle. When the target scene is a reverse driving scene, the target camera is a rear camera of the vehicle.

[0193] A car auxiliary line UI picture acquisition unit 22, used to acquire a configured car auxiliary line UI picture;

[0194] A turning angle obtaining unit 23, used to obtain the current front wheel turning angle of the vehicle;

[0195] The calibration position search unit 24 is used to search the target calibration position of the vehicle auxiliary line in the image taken by the target camera under the current front wheel turning angle in the configured vehicle auxiliary line calibration information file, and the correspondence between the different front wheel turning angles recorded in the vehicle auxiliary line calibration information file and the calibration positions of the vehicle auxiliary line in the images taken by the front and rear cameras is obtained by using the vehicle auxiliary line calibration method described in any of the above embodiments;

[0196] The image rendering unit 25 is used to render the vehicle auxiliary line UI picture into the image captured by the target camera according to the target calibration position, and output the rendered image for display.

[0197] In a possible implementation, the process in which the image rendering unit renders the auxiliary line UI picture into the image captured by the target camera according to the target calibration position includes:

[0198] The vehicle auxiliary line UI picture is rendered to the target calibration position in the image taken by the target camera through the configured rendering engine.

[0199] In a possible implementation, the process of rendering the auxiliary line UI picture to the target calibration position in the image captured by the target camera by the image rendering unit through the configured rendering engine includes:

[0200] Divide the texture coordinates of the vehicle auxiliary line UI image and the target calibration position into n parts with equal distances, and correspond them one by one to n groups of texture coordinates and vertex coordinates of the texture map;

[0201] Through the configured rendering engine, the vehicle auxiliary line UI picture is rendered into the image taken by the camera according to the correspondence between each set of texture coordinates and the vertex coordinates of the texture map.

[0202] In a possible implementation, the image rendering unit may also perform cropping and transformation processing on the rendered image before outputting and displaying the rendered image, and then outputting and displaying the processed image.

[0203] The present application also provides an electronic device in an embodiment. Fig. 9 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, vehicle-mounted terminals, servers, etc. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0204] like Fig. 9 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603 to implement the vehicle auxiliary line calibration method or vehicle auxiliary line display method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in RAM 603. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0205] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0206] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any vehicle auxiliary line calibration method or vehicle auxiliary line display method provided in an embodiment of the present application.

[0207] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any vehicle auxiliary line calibration method or vehicle auxiliary line display method provided in the embodiment of the present application.

[0208] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0209] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0210] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0211] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0212] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A method for calibrating a vehicle auxiliary line, characterized in that: include: According to the vehicle kinematic model, the position of the instantaneous rotation center on the straight line where the rear wheels of the vehicle are located is calculated at different front wheel turning angles of the vehicle; For each target structural part on the vehicle, calculating the distance from the position of the target structural part to the instantaneous rotation center, the target structural part being a designated vehicle structural part for which a vehicle auxiliary line needs to be drawn; An arc is drawn with the instantaneous rotation center as the center and the distance as the radius, and an arc of a set length located on a side of a first straight line away from the vehicle body is taken as a driving trajectory of the target structural component, wherein the first straight line is a straight line spaced from the outer contour of the vehicle by a set safety distance value; According to the position of the driving trajectory of the target structural part in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras, the calibrated position of the driving trajectory of the target structural part in the images taken by the front and rear cameras is determined.

2. The method according to claim 1, characterized in that: The target structure includes each wheel of the vehicle; The process of calculating the distance from the position of the target structural component to the instantaneous rotation center includes: Calculate the distances from the outer wheel point position and the inner wheel point position of each wheel to the instantaneous rotation center; The process of obtaining the driving trajectory of the target structural component includes: In the forward driving scenario, for each front wheel of the vehicle: with the instantaneous rotation center as the center, draw an arc with the distances from the outer wheel point position and the inner wheel point position of the front wheel to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain the outer wheel driving trajectory line and the inner wheel driving trajectory line of the front wheel, and the outer wheel driving trajectory line and the inner wheel driving trajectory line of the front wheel form the driving trajectory line of the front wheel; The process of determining the calibrated position of the driving trajectory of the target structural component in the image captured by the front camera includes: According to the position of the driving track line of the front wheel in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front camera, the calibrated position of the driving track line of the front wheel in the image taken by the front camera is determined.

3. The method according to claim 2, characterized in that The process of obtaining the driving trajectory of the target structural component also includes: In the reversing scenario, for each rear wheel of the vehicle: with the instantaneous rotation center as the center, draw an arc with the distances from the outer wheel point position and the inner wheel point position of the rear wheel to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain the outer wheel driving trajectory line and the inner wheel driving trajectory line of the rear wheel, and the outer wheel driving trajectory line and the inner wheel driving trajectory line of the rear wheel form the driving trajectory line of the rear wheel; The process of determining the calibrated position of the driving trajectory of the target structural component in the image captured by the rear camera includes: According to the position of the driving track line of the rear wheel in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the rear camera, the calibrated position of the driving track line of the rear wheel in the image taken by the rear camera is determined.

4. The method according to claim 1, characterized in that The target structural member includes an outer edge of a vehicle rearview mirror; The process of calculating the distance from the position of the target structural component to the instantaneous rotation center includes: Calculating the distance from the position of the outer edge of the vehicle rearview mirror to the instantaneous rotation center; The process of obtaining the driving trajectory of the target structural component includes: Draw an arc with the instantaneous rotation center as the center and the distance from the position of the outer edge of the vehicle rearview mirror to the instantaneous rotation center as the radius, and take an arc of a set length on the side of the first straight line away from the vehicle body to obtain a driving trajectory line of the outer edge of the vehicle rearview mirror; The process of determining the calibrated position of the driving trajectory of the target structural component in the images taken by the front and rear cameras includes: According to the position of the driving trajectory line of the outer edge of the vehicle rearview mirror in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras, the calibration position of the driving trajectory line of the outer edge of the vehicle rearview mirror in the images taken by the front and rear cameras is determined.

5. The method according to claim 1, characterized in that In a forward driving scenario, the first straight line is located in front of the vehicle, parallel to the front bumper and separated by a set safety distance value. In a reversing scenario, the first straight line is located behind the vehicle, parallel to the rear bumper and separated by a set safety distance value.

6. The method according to any one of claims 1 to 5, characterized in that: The conversion relationship between the world coordinate system and the image coordinate system captured by the front and rear cameras is calculated based on the extrinsic rotation matrix, extrinsic translation vector and intrinsic matrix of the front and rear cameras.

7. A method for displaying vehicle auxiliary lines, characterized in that: include: In response to a vehicle auxiliary line display instruction under a target scene, an image captured by a target camera is acquired, wherein when the target scene is a forward driving scene, the target camera is a front camera of the vehicle, and when the target scene is a reverse driving scene, the target camera is a rear camera of the vehicle; Get the configured vehicle auxiliary line UI image and the current front wheel steering angle of the vehicle; In the configured vehicle auxiliary line calibration information file, searching for the target calibration position of the vehicle auxiliary line in the image captured by the target camera under the current front wheel turning angle, the correspondence between the different front wheel turning angles recorded in the vehicle auxiliary line calibration information file and the calibration positions of the vehicle auxiliary line in the images captured by the front and rear cameras is obtained by using the vehicle auxiliary line calibration method according to any one of claims 1 to 6; The vehicle auxiliary line UI picture is rendered into the image captured by the target camera according to the target calibration position, and the rendered image is output and displayed.

8. The method according to claim 7, characterized in that The process of rendering the auxiliary line UI picture into the image captured by the target camera according to the target calibration position includes: The vehicle auxiliary line UI picture is rendered to the target calibration position in the image taken by the target camera through the configured rendering engine.

9. The method according to claim 8, characterized in that The process of rendering the auxiliary line UI picture to the target calibration position in the image captured by the target camera through the configured rendering engine includes: Divide the texture coordinates of the vehicle auxiliary line UI image and the target calibration position into n parts with equal distances, and correspond them one by one to n groups of texture coordinates and vertex coordinates of the texture map; Through the configured rendering engine, the vehicle auxiliary line UI picture is rendered into the image taken by the camera according to the correspondence between each set of texture coordinates and the vertex coordinates of the texture map.

10. The method according to claim 7, characterized in that Before the rendered image is output for display, it also includes: The rendered image is cropped and transformed, and the processed image is output and displayed.

11. A vehicle auxiliary line calibration device, characterized in that: include: An instantaneous rotation center determination unit is used to calculate the position of the instantaneous rotation center on the straight line where the rear wheels of the vehicle are located when the front wheels of the vehicle have different turning angles according to the vehicle kinematic model; A distance calculation unit, used for calculating the distance from the position of each target structural part on the vehicle to the instantaneous rotation center, wherein the target structural part is a designated vehicle structural part for which a vehicle auxiliary line needs to be drawn; an arc drawing unit, used for drawing an arc with the instantaneous rotation center as the center and the distance as the radius, taking an arc of a set length on a side of a first straight line away from the vehicle body as a driving trajectory of the target structural component, wherein the first straight line is a straight line spaced from the outer contour of the vehicle by a set safety distance value; A coordinate conversion unit is used to determine the calibrated position of the driving trajectory line of the target structural part in the images taken by the front and rear cameras according to the position of the driving trajectory line of the target structural part in the world coordinate system and the conversion relationship between the world coordinate system and the image coordinate system taken by the front and rear cameras.

12. A vehicle auxiliary line display device, characterized in that: include: An image acquisition unit, configured to acquire an image captured by a target camera in response to a vehicle auxiliary line display instruction in a target scene, wherein when the target scene is a forward driving scene, the target camera is a front camera of the vehicle, and when the target scene is a reverse driving scene, the target camera is a rear camera of the vehicle; A car auxiliary line UI picture acquisition unit is used to acquire the configured car auxiliary line UI picture; A turning angle acquisition unit, used to acquire the current front wheel turning angle of the vehicle; a calibration position search unit, used to search the target calibration position of the vehicle auxiliary line in the image taken by the target camera under the current front wheel turning angle in the configured vehicle auxiliary line calibration information file, wherein the correspondence between different front wheel turning angles recorded in the vehicle auxiliary line calibration information file and the calibration positions of the vehicle auxiliary line in the images taken by the front and rear cameras is obtained by using the vehicle auxiliary line calibration method according to any one of claims 1 to 6; An image rendering unit is used to render the vehicle auxiliary line UI picture into the image taken by the target camera according to the target calibration position, and output the rendered image for display.

13. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the vehicle auxiliary line calibration method according to any one of claims 1 to 6, or to implement the various steps of the vehicle auxiliary line display method according to any one of claims 7 to 10.

14. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the vehicle auxiliary line calibration method according to any one of claims 1 to 6 is implemented, or each step of the vehicle auxiliary line display method according to any one of claims 7 to 10 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the vehicle auxiliary line marking method according to any one of claims 1 to 6, or implements the steps of the vehicle auxiliary line display method according to any one of claims 7 to 10.

Images