Method, computer program product and device for calibrating superimposed auxiliary graphics in an image from a vehicle camera

Abstract

The present invention relates to a method, a computer program product, and a device for calibrating auxiliary graphics to be superimposed in an image from a camera (30) of a vehicle (40), wherein the method comprises: a first step for receiving an image from the camera (30) while the camera (30) is capturing an area surrounding the vehicle (40) in which a plurality of predefined markings (50) are arranged at predefined positions within a plane surrounding the vehicle (40), a second step for identifying the plurality of markings (50) and their position within the image from the camera (30), and a third step for calculating a transformation rule for at least one auxiliary graphic to be superimposed on the image from the camera (30) based on information about the local relationship of the camera (30) to the markings (50).so that, by applying the transformation rule to the at least one auxiliary graphic, a representation of the auxiliary graphic corresponding to a perspective of the camera (30) and the environment captured by the camera (30) can be determined, and a fourth step for superimposing the image of the camera (30) by means of the transformed representation of the at least one auxiliary graphic.

Description

State of the art

[0001] The present invention relates to a method, a computer program product and a device for calibrating auxiliary graphics to be superimposed in an image of a camera of a vehicle.

[0002] Various driver assistance systems for vehicles are known from the prior art, which are designed, among other things, to facilitate certain driving maneuvers such as reversing and / or parking by means of such auxiliary graphics by superimposing auxiliary graphics in images from one or more cameras (e.g. near-field cameras) of the vehicles.

[0003] For this purpose, it is necessary to adapt the auxiliary graphics to a perspective of the cameras on the surroundings of the vehicles so that the respective drivers of the vehicles can orient themselves directly using the superimposed auxiliary graphics during a driving maneuver.

[0004] For this purpose, during the development phase of such vehicles, manual calibrations of the auxiliary graphics are carried out on the basis of auxiliary lines arranged in the vehicle's surroundings. The vehicle is placed in a predefined position and with a predefined orientation with respect to the auxiliary lines and the parameters of the auxiliary graphics are then manually adjusted in order to be able to insert the auxiliary graphics into the camera images with the correct perspective. Disclosure of the invention

[0005] According to a first aspect of the present invention, a method is proposed for calibrating auxiliary graphics to be superimposed in an image of a camera of a vehicle, wherein the vehicle is preferably a road vehicle such as a car, a van, a bus or a truck, without being restricted to such vehicle types.

[0006] The camera is, for example, a front camera, a rear view camera, an all-round view camera or a different surroundings detection camera of the vehicle, whereby the position of the camera on the vehicle and the orientation of the camera with respect to the surroundings are generally not restricted.

[0007] It should also be noted that, prior to executing the method according to the invention, the camera is preferably in a calibrated state in which a predefined arrangement position of the camera on the vehicle and a predefined orientation of the camera with respect to the surroundings of the vehicle are set. This can be achieved, in particular, by performing a calibration process for the camera, which is carried out, for example, during vehicle production. This ensures that respective cameras of vehicles of the same and / or a similar vehicle model have an identical starting point with regard to the camera's perspective of the surroundings.

[0008] The image from the camera is output, for example, in a central display of an on-board computer system of the vehicle and / or in a display of an instrument cluster and / or in a head-up display of the vehicle and / or in a different display device of the vehicle.

[0009] In a first step of the method according to the invention, an image from the camera is received while the camera captures an area surrounding the vehicle in which a plurality of predefined markings are arranged at predefined positions within a plane surrounding the vehicle (preferably a road plane).

[0010] The image is received, for example, by means of an evaluation unit according to the invention, which can be a component of the vehicle and / or a component not belonging to the vehicle, such as a computer, a tablet computer, a cloud server, etc., and which is connected wirelessly and / or wired, directly and / or indirectly, to the camera via information technology. Accordingly, it is also conceivable for the evaluation unit to be composed of a first evaluation unit, which is a component of the vehicle, and a second evaluation unit, which is not belonging to the vehicle. Such an evaluation unit can also advantageously be used to carry out the method steps described below.

[0011] Preferably, the predefined positions of the markers are set in such a way that an adjustment of parameters that define the appearance of the auxiliary graphic is made possible in a particularly simple manner. In an exemplary case in which the auxiliary graphic represents one or more trajectories whose lines are divided into sections of different colors, the markers are advantageously positioned at positions in the environment where the respective sections adjoin one another, in order to be able to define the color changes between the individual sections at these positions.

[0012] In a second step of the method according to the invention, the plurality of markers and their respective positions within the camera image are identified. For this purpose, image analysis methods known from the prior art are used, for example.

[0013] In a third step of the method according to the invention, a transformation rule for at least one auxiliary graphic to be superimposed on the camera image is calculated based on information about a spatial relationship of the camera to the markings, such that by applying the transformation rule to the at least one auxiliary graphic, which is predefined, for example, in the form of a basic configuration, a representation of the auxiliary graphic corresponding to a perspective of the camera and the surroundings captured by the camera can be determined. In other words, the transformation rule makes it possible to transform the auxiliary graphic predefined in a basic configuration such that it corresponds to a perspective of the camera on the surroundings of the camera or the vehicle. It should be noted that such a basic configuration can, in principle, be designed in any desired way.For example, the basic configuration can be defined by one or more functions describing the auxiliary graphic and / or by parameters that define support points and / or optical properties of the auxiliary graphic and / or individual sub-areas of the auxiliary graphic.

[0014] The information about the local reference is stored, for example, in a memory unit of the vehicle (e.g. in a memory unit of the camera and / or the evaluation unit, etc.) and / or is transmitted to the vehicle, in particular to the evaluation unit according to the invention, for carrying out the method according to the invention.

[0015] It should be noted that the transformation rule can consist of a large number of individual transformation rules, each of which is intended for a specific part of the auxiliary graphic.

[0016] In a fourth step of the method according to the invention, the calculated transformation rule is provided. For this purpose, the transformation rule is transmitted, for example, via a wireless and / or a wired communication connection to a component of the vehicle, which is configured to store the transformation rule in a memory unit. Alternatively or additionally, the transformation rule can be provided to a component away from the vehicle, which is used, for example, in a production facility for manufacturing the vehicle, in order to incorporate the transformation rule into the vehicle during or at the end of a production line for later use. In this way, it is advantageously possible to transfer the determined transformation rule, which can be represented in the form of data, to other vehicles (e.g., of the same and / or a similar model).

[0017] The method according to the invention offers, among other advantages, that the calibration process for the auxiliary graphics can be automated and thus carried out more quickly and / or accurately. This can shorten development times for affected driver assistance systems and / or facilitate and / or improve the implementation of such calibration by aftermarket manufacturers of vehicle cameras.

[0018] The subclaims show preferred developments of the invention.

[0019] Preferably, the transformation rule is then used to overlay the transformed representation of the at least one auxiliary graphic onto the camera image, for example, and output it in one of the vehicle's display devices described above. It is understood that a plurality of auxiliary graphics can be adapted according to the method according to the invention and output together in the camera image.

[0020] In an advantageous embodiment of the present invention, at least three, preferably four, and more preferably more than four predefined markers are used, so that a particularly reliable calculation basis for the perspective transformation of the auxiliary graphic can be provided. Particularly advantageously, the plurality of markers represents respective corners of one or more polygons, thereby facilitating the calculation of the position of the respective markers in space, since the polygon shape spannes a surface (e.g., a triangular surface) whose position within the camera image can be reliably determined.

[0021] Particularly advantageously, the information about the camera's spatial relationship to the plurality of markers is obtained by positioning the vehicle at a predefined position and with a predefined orientation relative to the markers, and by providing information about the camera's perspective on the surroundings at this predefined position and orientation of the vehicle. The predefined position and orientation for the vehicle can be achieved, for example, by parking markers for the vehicle in the vicinity of the markers, which can be achieved by means of a manual and / or automatic parking process.

[0022] Alternatively or additionally, the information about the spatial relationship of the camera to the plurality of markers is automatically determined based on the plurality of markers themselves and / or on additional auxiliary lines provided in the plane surrounding the vehicle. Such additional auxiliary lines are designed, for example, in the form of a grid, a circle, a plurality of concentrically arranged circles, etc., so that a position and orientation of the vehicle or the camera with respect to these auxiliary lines can be determined using image analysis software capable of determining a camera perspective based on a perspective distortion of the auxiliary lines (the shape and / or arrangement of which are known to the image analysis software).This offers the advantage that the vehicle does not have to be positioned exactly at a predefined parking position in order to perform the calibration of the auxiliary graphics, thus saving time and costs.

[0023] Furthermore, it is possible for the markings to be identical to one another or at least partially different from one another. An identical design of the markings can simplify the automatic identification of the markings in the camera image, while at least partially different markings can facilitate determining the position and orientation of the vehicle with respect to the markings.

[0024] In a particularly advantageous embodiment of the present invention, the markings are each designed as geometric figures (e.g., a circle, a triangle, a rectangle, a square, an ellipse, etc.) and / or color-coded to clearly distinguish the respective markings from one another. Alternatively or additionally, the markings are designed as a two-dimensional code, which enables identification of the markings and / or contains additional information about the markings.

[0025] In an advantageous case in which the markings are designed as a two-dimensional code (e.g., as a binary code), the two-dimensional code is preferably an ArUco and / or a QR code (both codes are known from the prior art). Alternatively or additionally, the two-dimensional code comprises information about a function and / or a position of the respective marking. Information about a function of a respective marking is, for example, information that represents a position of the respective markings relative to one another and / or to a higher-level reference system. Alternatively or additionally, the information about the function represents, for example, a specification for parameterizing the auxiliary graphic, such as a color and / or line width and / or texture to be used for the auxiliary graphic or a section of the auxiliary graphic, etc.

[0026] In an advantageous embodiment of the present invention, the at least one auxiliary graphic represents a travel path and / or a trajectory for the vehicle. Such an auxiliary graphic is superimposed, for example, on an image from a rearview camera of the vehicle to facilitate reversing and / or reverse parking, etc. for a driver. For example, such distinctive auxiliary graphics are superimposed on the camera image in such a way that they extend from the rear of the vehicle into the surrounding area.

[0027] Advantageously, parameters for adjusting the optical properties of the at least one auxiliary graphic are determined based on the plurality of markings. For example, the markings can be used to mark positions where the auxiliary line transitions from one color to the next, between which stretching of certain areas of the auxiliary graphic is required, etc.

[0028] According to a second aspect of the present invention, a computer program product is proposed for calibrating auxiliary graphics to be superimposed in an image from a camera of a vehicle, wherein the computer program product has instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the first aspect of the invention. The computer program product is stored, for example, on any data storage medium (e.g., a hard disk, a volatile memory, a non-volatile memory, an optical storage device, etc.). The features, combinations of features, and the advantages resulting from them correspond to those explained in connection with the first aspect of the invention, so that reference is made to the above explanations to avoid repetition.

[0029] According to a third aspect of the present invention, a device is proposed for calibrating auxiliary graphics to be superimposed in a camera image of a vehicle, wherein the vehicle, as mentioned above, is preferably a road vehicle such as a car, a van, a bus, or a truck. The device has an evaluation unit, which is designed, for example, as an ASIC, FPGA, processor, digital signal processor, microcontroller, or the like, and can be a component of the vehicle and / or a component not belonging to the vehicle. The evaluation unit is configured to receive an image from the camera while the camera captures an area surrounding the vehicle in which a plurality of predefined markers are arranged at predefined positions within a plane surrounding the vehicle.The evaluation unit is further configured to identify the plurality of markers and their position within the camera image and to calculate a transformation rule for at least one auxiliary graphic to be superimposed on the camera image on the basis of information about a spatial relationship of the camera to the markers, so that by applying the transformation rule to the at least one auxiliary graphic, a representation of the auxiliary graphic corresponding to a perspective of the camera and the environment captured by the camera can be determined.

[0030] According to a fourth aspect of the present invention, a use of the transformation rule determined according to the first aspect of the invention is proposed for overlaying a camera image using a representation of the at least one auxiliary graphic transformed by the transformation rule. Such use occurs, for example, in mass-produced vehicles in which the transformation rule is stored in the form of data for later use. Short description of the drawings

[0031] Embodiments of the invention are described in detail below with reference to the accompanying drawings. In the drawing: Fig. 1 a schematic view of a device according to the invention, a vehicle and markings according to the invention for calibrating auxiliary graphics to be superimposed in an image of a camera of the vehicle; and Fig. 2 a schematic view of a display with an image according to the invention from a camera of a vehicle with auxiliary graphics superimposed according to the invention. Embodiments of the invention

[0032] Fig. 1 shows a schematic view of a device according to the invention, a vehicle 40 and markings 50 according to the invention for calibrating auxiliary graphics to be superimposed in an image of a camera 30 of the vehicle 40.

[0033] The vehicle 40, which is designed here as a passenger car, has a camera 30 designed as a rear-view camera, which is arranged at the rear of the vehicle 40 and is directed towards an area surrounding the vehicle 40, opposite to the main direction of travel of the vehicle 40.

[0034] The camera 30 is connected in terms of information technology to a first evaluation unit 70 of the vehicle 40, which is configured to receive an image from the camera 30, while the camera 30 captures an area surrounding the vehicle 40 in which a plurality of predefined markings 50 are arranged at predefined positions within a plane surrounding the vehicle 40.

[0035] The first evaluation unit 70 is further configured to transmit the image of the camera 30 via a wireless communication unit 80 of the vehicle 40 to a second evaluation unit 75, which here represents a computer arranged away from the vehicle 40.

[0036] The second evaluation unit 75 is configured to identify the plurality of markings 50 and their position within the image of the camera 30. For this purpose, the markings are each configured as two-dimensional ArUco codes, which enable a unique identification of the respective markings 50.

[0037] The second evaluation unit 75 is further configured to calculate a transformation rule for an auxiliary graphic to be superimposed on the image of the camera 30 on the basis of information about a spatial relationship of the camera 30 to the markings 50, so that by applying the transformation rule to the one auxiliary graphic, a representation of the auxiliary graphic 10 corresponding to a perspective of the camera 30 and the environment captured by the camera 30 can be determined.

[0038] The information about the spatial relationship of the camera 30 to the plurality of markings 50 is obtained here by placing the vehicle 40 at a predefined position and with a predefined orientation with respect to the markings 50, and by providing information about the perspective of the camera 30 on the surroundings at this predefined position and orientation of the vehicle 40 in a storage unit (not shown) of the second evaluation unit 75. The predefined position and the predefined orientation of the vehicle 40 are ensured by placing the vehicle 40 within a parking marking 65.

[0039] Additional auxiliary lines 60 provided in the vicinity of the vehicle 40 are provided here for more precise positioning of the vehicle 40.

[0040] Finally, the second evaluation unit 75 is configured to transmit the calculated transformation rule in the form of data to the vehicle 40 and, if applicable, to other identically or similarly designed vehicles, so that the transformation rule can subsequently be used in the vehicle 40 and / or in the other vehicles.

[0041] The use is preferably such that the image of the camera 30 is output in a display 90 of the vehicle 40 by means of a control by the first evaluation unit 70, wherein a representation of the auxiliary graphic transformed on the basis of the transformation rule is superimposed on the image of the camera 30.

[0042] Fig. Figure 2 shows a schematic view of a display 90 of a vehicle 40 (see Fig. 1) with an image 20 according to the invention from a camera 40 (see also Fig. 1) of the vehicle 40 with auxiliary graphics 10 superimposed according to the invention.

[0043] The auxiliary graphics 10 are designed here as respective trajectories for the wheels of the vehicle 40, which can be adjusted depending on a current steering angle of the vehicle 40. Furthermore, the markings 50 are shown to explain that the position of the markings 50 in the image 20 of the camera 30 is used to color individual sections of the respective markings 50 differently to indicate specific distance ranges of the trajectories. The individual sections are identified here by different hatching of the trajectory lines.

Claims

[1] Method for calibrating auxiliary graphics (10) to be superimposed in an image (20) of a camera (30) of a vehicle (40), comprising: - a first step of receiving an image (20) from the camera (30) while the camera (30) captures an area surrounding the vehicle (40) in which a plurality of predefined markers (50) are arranged at predefined positions within a plane surrounding the vehicle (40), - a second step for identifying the plurality of markers (50) and their position within the image (20) of the camera (30), - a third step for calculating a transformation rule for at least one auxiliary graphic (10) to be superimposed on the image (20) of the camera (30) on the basis of information about a spatial relationship of the camera (30) to the markings (50), so that by applying the transformation rule to the at least one auxiliary graphic (10), a representation of the auxiliary graphic (10) corresponding to a perspective of the camera (30) and the environment captured by the camera (30) can be determined, and - a fourth step to provide the calculated transformation rule. [2] The method according to claim 1, further comprising superimposing the image (20) of the camera (30) by means of the transformed representation of the at least one auxiliary graphic (10). [3] Method according to one of the preceding claims, wherein - at least three, preferably four and more preferably more than four predefined markings (50) are used, and / or - the plurality of markers (50) represent respective corners of one or more polygons. [4] Method according to one of the preceding claims, wherein the information about the spatial relationship of the camera (30) to the plurality of markers (50) is obtained by placing the vehicle (40) at a predefined position and with a predefined orientation with respect to the markers (50) and providing information about the perspective of the camera (30) on the surroundings at this predefined position and orientation of the vehicle (40). [5] Method according to one of the preceding claims, wherein the information about the spatial relationship of the camera (30) to the plurality of markings (50) is automatically determined on the basis of the plurality of markings (50) themselves and / or on the basis of further auxiliary lines (60) provided in the plane surrounding the vehicle (40). [6] Method according to one of the preceding claims, wherein the markings (50) are identical to one another or at least partially different from one another. [7] Method according to one of the preceding claims, wherein the markings (50) each - are designed as geometric figures, and / or - are color-coded, and / or - are designed as a two-dimensional code which enables identification of the markings (50) and / or contains additional information about the markings (50). [8] A method according to claim 7, wherein the two-dimensional code - is an ArUco and / or a QR code, and / or - has information about a function and / or a position of the respective marking (50). [9] Method according to one of the preceding claims, wherein the at least one auxiliary graphic (10) represents a travel path and / or a trajectory for the vehicle (40). [10] Method according to one of the preceding claims, wherein parameters for adapting optical properties of the at least one auxiliary graphic (10) are determined on the basis of the plurality of markings (50). [11] Computer program product for calibrating auxiliary graphics (10) to be superimposed in an image (20) of a camera (30) of a vehicle (40), comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to one of the preceding claims. [12] Device for calibrating auxiliary graphics (10) to be superimposed in an image (20) of a camera (30) of a vehicle (40), comprising an evaluation unit (70), wherein the evaluation unit (70) is arranged - to receive an image (20) of the camera (30) while the camera (30) captures an area surrounding the vehicle (40) in which a plurality of predefined markers (50) are arranged at predefined positions within a plane surrounding the vehicle (40), - to identify the plurality of markings (50) and their position within the image (20) of the camera (30), - to calculate a transformation rule for at least one auxiliary graphic (10) to be superimposed on the image (20) of the camera (30) on the basis of information about a spatial relationship of the camera (30) to the markings (50), so that by applying the transformation rule to the at least one auxiliary graphic (10), a representation of the auxiliary graphic (10) corresponding to a perspective of the camera (30) and the environment captured by the camera (30) can be determined. [13] Use of the transformation rule determined according to one of claims 1 to 10 for superimposing an image (20) of a camera (30) by means of a representation of the at least one auxiliary graphic (10) transformed by the transformation rule.

Images