Displaying image data inside a vehicle using a changing projection surface

Abstract

To display the image data, initial and final projection planes (9a, 9b) are provided, and the video stream is received and buffered in a buffer storage that is updated at a frame rate. The image data is read from the buffer at a read rate in an initial read instance, intermediate read instances, and final read instances. Respective sets of initial and final sampling points (11a, 11b) on the projection planes (9a, 9b) are determined. For each intermediate read instance, intermediate sampling points (12, 13) that interpolate between the final sampling point (11b) and the respective initial sampling point (11a), and intermediate projection planes (10) including the intermediate sampling points (12, 13) are determined. The read image data is projected onto the respective intermediate projection planes (10), and corresponding images are displayed on the display device (5).

Description

[Technical Field]

[0001] The present invention is directed to a computer-implemented method for displaying image data in a vehicle, an electric vehicle guidance system for a vehicle, a computer program, and a computer-readable storage medium. [Background technology]

[0002] A vehicle camera system, e.g., a vehicle surround view system, which may include one or more cameras positioned at different locations on the vehicle, may be used for driver assistance functions or other functions for autonomous or semi-autonomous driving. One camera image, or two or more camera images from different cameras stitched together into a combined view, may be projected onto a predetermined projection surface that is part of a two-dimensional manifold in three-dimensional space, such as a bowl shape or the like. Furthermore, this one projected image or multiple projected images may be transformed according to the respective viewing parameters of a virtual observer, also referred to as a virtual camera, so that the images appear as if they were observed by a virtual observer or captured by a virtual camera, respectively. The position and / or orientation of the virtual observer may be set or modified, for example, by a user or automatically by the vehicle.

[0003] The application note, "360° Wrap-Around Video Imaging Technology Ready for Integration with Fujitsu Graphics SoCs" (available September 12, 2022, at the URL https: / / www.fujitsu.com / us / imagesgig5 / 360_OmniView_AppNote.pdf), describes a video imaging technology that enables a 360-degree view of the vehicle's perimeter in real time.

[0004] German Patent No. 102015105529A1 relates to a method for transforming an image representing an area surrounding a motor vehicle from the viewpoint of a virtual camera, the image being represented by a transformation from a number of real images generated by means of a number of real cameras of the motor vehicle.

[0005] US 2021 / 0125401A1 relates to a method for representing an environmental region of a motor vehicle in an image, where actual images of the environmental region are captured by a real camera on the motor vehicle and an image is generated from these actual images. The image is represented from the viewpoint of a virtual camera within the environmental region and the image is generated as a bowl shape.

[0006] If the position and / or orientation of the virtual observer is changed while the video stream is being displayed, the shape of the projection surface may also change. The change may be implemented as a hard switch, i.e., the projection surface is changed from an initial shape to a final shape without a transition. However, the change in the projection surface is not necessarily related to a change in the position and / or orientation of the virtual observer and may result, for example, from a change in user settings. Summary of the Invention [Problem to be solved by the invention]

[0007] In view of this, it is an object of the present invention to provide the possibility of achieving a smooth transition from one projection plane to another. [Means for solving the problem]

[0008] This object is achieved by the respective subject matter of the independent claims. Further implementations and preferred embodiments are the subject matter of the dependent claims.

[0009] The invention is based on the idea of ​​sampling an initial projection plane and a final projection plane, and determining a respective intermediate projection plane for each of a number of successive intermediate read instances for reading a buffer storage for buffering a video stream, each intermediate projection plane comprising intermediate sampling points that interpolate between sampling points on the final projection plane and corresponding sampling points on the initial projection plane.

[0010] According to an aspect of the present invention, a computer-implemented method for displaying image data in a vehicle, particularly an automobile, is provided. A predetermined initial projection plane and a predetermined final projection plane are provided. A video stream depicting the vehicle's environment, particularly the external environment, is received from a camera system of the vehicle. The video stream is buffered in a buffer storage, and the buffer storage is updated at a predetermined frame rate. The image data, particularly each updated image data, is read from the buffer storage at a predetermined read rate. The reading occurs at an initial read instance, multiple successive intermediate read instances following the initial read instance, and a final read instance following the multiple intermediate read instances. A set of initial sampling points on the initial projection plane and final sampling points on the final projection plane are determined. Each final sampling point in the set of final sampling points is assigned to only one of the initial sampling points in the set of initial sampling points.

[0011] For each intermediate readout instance, a set of intermediate sampling points is determined, and each intermediate sampling point of the set of intermediate sampling points interpolates between one of the final sampling points and a respective assigned initial sampling point. For each intermediate readout instance, an intermediate projection plane including the respective set of intermediate sampling points is determined. For each intermediate readout instance, the read image data is projected onto the respective intermediate projection plane, and an intermediate image corresponding to the projected image data is displayed on a display device of the vehicle.

[0012] Unless otherwise specified, all steps of the computer-implemented method may be performed by at least one computing device, which may also be referred to as a data processing device, in particular a vehicle. In particular, the at least one computing device includes at least one processing circuit configured or adapted to perform the steps of the computer-implemented method. To this end, the at least one computing device may, for example, store a computer program including instructions that, when executed by the at least one computing device, cause the at least one computing device to perform the computer-implemented method. It should be understood that the step of displaying each intermediate image on a display device is such that the at least one computing device controls the display device to display each intermediate image. In this case, the display is not necessarily part of the at least one computing device. Alternatively, the display may be part of the at least one computing device.

[0013] A computing device may be understood as, among other things, a data processing device including processing circuitry. Thus, a computing device may, among other things, process data to perform computing operations, which may also include operations to perform index accesses to data structures, for example, look-up tables (LUTs).

[0014] In particular, a computing device may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more systems-on-chips (SoCs). A computing device may also include one or more processors, such as one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). A computing device may also include a physical or virtual cluster of computers or other of the above devices.

[0015] In various embodiments, a computing device includes one or more hardware and / or software interfaces and / or one or more memory units.

[0016] The memory unit may be implemented as a volatile data memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), or a non-volatile data memory, for example, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or flash EEPROM, a ferroelectric random access memory (FRAM), a magnetoresistive random access memory (MRAM), or a phase-change random access memory (PCRAM).

[0017] The initial and final projection planes may be provided, for example, by storing them on a memory unit of at least one computing device, in particular by storing corresponding mathematical descriptions, look-up tables, or another computer-readable data set.

[0018] A video stream includes multiple consecutive frames, each containing respective image data, which are stored in a buffer storage in sequence. In particular, when the respective image data of a frame is stored in the buffer storage, the respective image data of the preceding frame may be deleted or overwritten. However, the buffer storage may also be a ring buffer. In either case, the latest image data stored in the buffer storage changes at a predetermined frame rate and is read out at a predetermined readout rate. Thus, the reverse readout rate defines the time difference between the initial readout instance and the first intermediate readout instance, between all pairs of successive intermediate readout instances, and between the last intermediate readout instance and the final readout instance. The frames do not necessarily constitute the entire video stream, but may be part of it.

[0019] For example, for an initial readout instance, the readout image data may be projected onto an initial projection surface, and an initial image responsive to the projected image data is displayed on the display device, particularly before any of the intermediate images are displayed. Alternatively, or in addition, for a final readout instance, the readout image data is projected onto a final projection surface, and a final image responsive to the projected image data is displayed on the display device, particularly after all of the intermediate images have been displayed.

[0020] In particular, each projected image data according to the intermediate readout instance may be directly displayed on a display device or may be further processed before being displayed on a display device. In other words, an intermediate image may be obtained directly by each projected image data or by further processed projected image data. The further processing may include, for example, filtering or processing steps to stitch together image data from different cameras, transforming image data according to the viewing position and / or orientation or other viewing parameters of the virtual observer, etc.

[0021] The projection plane may be considered, for example, as a two-dimensional manifold in three-dimensional space. For example, the projection plane may be defined in a Cartesian coordinate system having coordinates X, Y, and Z using a function f, where Z=f(X,Y). However, the projection plane is not necessarily describable using such a function. In particular, in some implementations, the projection plane may also be defined in a piecewise manner using different functions for different parts of the three-dimensional space. The projection plane may be defined in a vehicle coordinate system, in particular, in which the position and orientation of the vehicle are known, in particular, fixed. For example, a designated reference point of the vehicle may be located at the origin of the coordinate system.

[0022] The projection planes may be portions of a two-dimensional manifold. Consequently, different projection planes may correspond to different portions of the same two-dimensional manifold. Alternatively, different projection planes may equally correspond to portions of different two-dimensional manifolds, e.g., defined by different functions f(X,Y).

[0023] The intermediate projection surfaces can be constructed in various ways given the respective intermediate sampling points. For example, they can be obtained by respective polygon meshes, with the intermediate sampling points representing the corners of the polygons. The intermediate projection surfaces can also be obtained by respective spline surfaces with the intermediate sampling points as control points. The intermediate projection surfaces can also be obtained by other surfaces that interpolate between the intermediate sampling points.

[0024] The number of final sampling points may be less than or equal to the number of initial sampling points. The number of intermediate sampling points for a given intermediate readout instance is equal to the number of final sampling points. The intermediate projection planes account for a smooth transition from the initial projection plane to the final projection plane, and for each intermediate readout instance, a respective intermediate image is displayed. As a result, the resulting sequence of displayed images accounts for a smooth transition from the initial view to the final view. Thus, a vehicle user looking at the display device will not see a sudden change in the displayed image due to a sudden change in the projection plane.

[0025] In particular, the initial, intermediate, and final projection planes may all be different from one another, and thus the displayed image changes for each readout instance.

[0026] Since a respective intermediate image is displayed for each intermediate readout instance, the readout rate is in particular equal to or less than the refresh rate of the display device, and preferably equal to the refresh rate. Note that the readout rate may be greater than the frame rate. In other words, each readout image data may remain unchanged for two or more consecutive intermediate readout instances. Nevertheless, the displayed image still generally changes because the intermediate projection surface generally changes. However, in other implementations, the readout rate may also be equal to or less than the frame rate, which means that the readout image data is different for each pair of subsequent intermediate readout instances.

[0027] According to some implementations, predetermined initial viewing parameters of the virtual observer, including an initial position of the virtual observer, and predetermined final viewing parameters of the virtual observer, including a final position of the virtual observer, are provided and stored, in particular, in a memory unit of at least one computing device. For each intermediate readout instance, intermediate viewing parameters of the virtual observer, including respective intermediate positions of the virtual observer, are determined. Here, the intermediate positions are on a predetermined curve connecting the initial position to the final position. For each intermediate readout instance, the readout image data is projected onto a respective intermediate projection plane and transformed according to the respective intermediate viewing parameters. Each intermediate image, for example, depends on and corresponds to the projected and transformed image data.

[0028] In such an implementation, a change in the projection plane from the initial projection plane through intermediate projection planes to the final projection plane is accompanied by a change in the position of the virtual observer. The change in the position of the virtual observer occurs as a smooth transition. However, the dynamics of the change in the projection plane do not necessarily match the dynamics of the change in the position of the virtual observer. In other words, the position of the virtual observer may remain unchanged for two or more consecutive readout instances, i.e., the initial, intermediate, and final readout instances.

[0029] Apart from the position of the virtual observer, the viewing parameters may include, for example, the orientation of the virtual observer. The viewing parameters may also include or define, in particular, the field of view of the virtual observer, also referred to as the vehicle camera. The viewing parameters may also include virtual mapping parameters that describe the mapping function of the virtual camera.

[0030] The virtual observer may be located anywhere within the vehicle's environment. Depending on the location of the virtual observer and / or further viewing parameters, the size of the projection surface may vary. In particular, the image data displayed on the display device appears as if they were captured by a virtual camera or viewed by a virtual observer, respectively.

[0031] The curve with the intermediate positions can be a straight line connecting the initial position to the final position or any other curve, and a user looking at the displayed image will have the impression that the observation point is flying along the curve.

[0032] According to some implementations, the initial viewing parameters include an initial orientation of the virtual observer, and the final viewing parameters include a final orientation of the virtual observer. For each intermediate readout instance, the respective intermediate viewing parameters include a respective intermediate orientation of the virtual observer.

[0033] The orientation of the virtual observer can be obtained, for example, by three orientation angles, e.g., Euler angles, and the intermediate orientations can therefore be considered to lie on respective curves in the parameter space of orientation angles connecting the initial orientation to the final orientation.

[0034] According to some implementations, the initial viewing parameters include an initial field of view of the virtual observer, and the final viewing parameters include a final field of view of the virtual observer, and for each intermediate readout instance, the respective intermediate viewing parameters include a respective intermediate field of view of the virtual observer.

[0035] The field of view of the virtual observer can be defined by two viewing angles, e.g., a horizontal angle and a vertical angle, and the intermediate fields of view can be considered to lie on respective curves in the parameter space of viewing angles connecting the initial field of view to the final field of view.

[0036] In some implementations, for each final sampling point, intermediate sampling points that interpolate between the final sampling point of all intermediate read instances and the assigned initial sampling point are on a straight line connecting the final sampling point and the assigned initial sampling point.

[0037] In this way, the shortest connection between each final and initial point can be used for interpolation, and therefore smaller intermediate projection planes may be needed for smooth transitions.

[0038] According to some implementations, for each final sampling point, the respective distances between the intermediate sampling points that complement the final sampling point and the assigned initial sampling point are the same for each pair of consecutive intermediate read instances.

[0039] In particular, the distance between the initial sampling point and the first intermediate sampling point, as well as the distance between the final sampling point and the last intermediate sampling point, are also equal to the distance between successive pairs of intermediate sampling points.

[0040] In other words, the transition is performed in a linear manner, which reduces the amount of calculations for calculating the intermediate projection planes.

[0041] In some implementations, the readout rate is greater than the frame rate.

[0042] As a result, the same image data may be read from the buffer storage for at least two consecutive intermediate read instances, so the transition appears smoother to an observing user.

[0043] Preferably, the readout rate is at least twice the frame rate, for example at least five times the frame rate. For example, the ratio of readout rate to frame rate may be in the interval [5, 20].

[0044] As a result, for each camera frame, the same buffered image data is read out at least two times, or at least five times, or at least the number of times given by the above ratio in each successive intermediate read instance, so that the transition appears smoother to an observing user.

[0045] In some implementations, the total number of intermediate readout instances can be adjusted to the above ratio so that the initial readout instance, all intermediate readout instances, and the final readout instance occur within a single frame and the image data read out for each is the same. Thus, the entire transition can occur within a single frame, which makes the transition appear even smoother. However, the total number of intermediate readout instances can also be adjusted to the above ratio so that the initial readout instance, all intermediate readout instances, and the final readout instance occur within a predetermined maximum number of frames.

[0046] According to some implementations, the initial projection plane is obtained by points that satisfy the following equation:

[0047]

number

[0048] In particular, the initial sampling points satisfy the following formula: where X, Y, and Z denote the Cartesian coordinates of each point on the initial projection plane, Z corresponds to the height above a predetermined ground plane where the vehicle is located, n is an even integer equal to or greater than 4, and a i , b i , and c is a predetermined real coefficient.

[0049] In other words, the initial projection plane is obtained by an n-th degree polynomial in two variables X and Y. Preferably, n=4. As a result, the shape of the initial projection plane can be depicted as a bowl, which has a relatively flat portion in the area where the vehicle is located and a relatively steep rise far away from the vehicle. In particular, the vehicle is centered at the origin of the coordinate system, and the XY plane corresponds to the ground plane. This represents a good approximation for the distance of the depicted object.

[0050] For n=4, the above formula for Z can be written as: Z=W[A(XX s ) 4 +B(YY s ) 4 ]-C.

[0051] In some implementations, the same is true for the final projection plane, in an analogous manner. In other words, in such implementations, the final projection plane is obtained by the points that satisfy the following equation:

[0052]

number

[0053] In particular, the final sampling points satisfy this equation: where X, Y, and Z denote the Cartesian coordinates of each point on the final projection plane, Z corresponds to the height above the ground plane, n' is an even integer equal to or greater than 4, and a i ', b i n′, and c′ are predetermined real coefficients. Preferably, n′=n.

[0054] It should be noted that the intermediate projection planes, and in particular the intermediate sampling points of a given intermediate projection plane, do not necessarily satisfy such an equation for a bivariate polynomial.

[0055] According to some implementations, the initial projection surface includes an initial base portion obtained by an initial portion of the contact plane on which the vehicle is located, the initial base portion within which the vehicle is located, and an initial raised portion adjacent to the initial base portion at an outer boundary of the initial base portion. Alternatively, or in addition, the final projection surface includes a final base portion obtained by a final portion of the contact plane, the final base portion within which the vehicle is located, and a final raised portion adjacent to the final base portion at an outer boundary of the final base portion.

[0056] In particular, the initial base portion and / or the final base portion are convex, or in other words, the initial base portion and / or the final base portion or their respective outer boundaries are convex geometric figures, in particular convex polygons. In contrast to the respective base portions, points on the respective raised portions have a non-zero height above the ground plane.

[0057] In some implementations, the described base and raised portion structure may also be used for intermediate projection surfaces.

[0058] For use cases or usage situations that may arise in the computer-implemented method and that are not explicitly described herein, it may be provided that error messages and / or prompts for user feedback are output and / or default settings and / or predetermined initial states are set in accordance with the method.

[0059] According to a further aspect of the present invention, there is provided an electric vehicle guidance system for a vehicle. The electric vehicle guidance system includes at least one computing device, particularly for the vehicle, that stores a predetermined initial projection plane and a predetermined final projection plane, particularly in one or more memory units of the at least one computing device. The at least one computing device is configured to receive a video stream depicting the vehicle's environment from a camera system of the vehicle and to buffer the video stream, for example, in a buffer storage of the at least one computing device, where the buffer storage is updated at a predetermined frame rate. The at least one computing device is configured to read image data from the buffer storage at the predetermined read rate in an initial read instance, multiple successive intermediate read instances, and a final read instance.

[0060] The at least one computing device is configured to determine a set of initial sampling points on the initial projection plane and final sampling points on the final projection plane, where each final sampling point is assigned to only one of the initial sampling points. The at least one computing device is configured to determine, for each intermediate readout instance, a set of intermediate sampling points, where each intermediate sampling point interpolates between one of the final sampling points and its assigned initial sampling point, and to determine an intermediate projection plane including the set of intermediate sampling points. The at least one computing device is configured, for each intermediate readout instance, to control a display device of the vehicle to project the read image data onto the respective intermediate projection plane and to display an intermediate image corresponding to the projected image data.

[0061] An electric vehicle guidance system may be understood as an electronic system configured to guide a vehicle in a fully automated or autonomous manner, and in particular, without requiring manual intervention or control by the driver or user of the vehicle. The vehicle performs all necessary functions, such as steering, decelerating, and / or accelerating, as well as monitoring and recording road traffic and automatically responding accordingly. In particular, an electric vehicle guidance system may implement a fully automated or fully autonomous driving mode in accordance with Level 5 of the SAE J3016 classification. An electric vehicle guidance system may also be implemented as an advanced driver assistance system (ADAS) that assists the driver for partially automated or partially autonomous driving. In particular, an electric vehicle guidance system may implement a partially automated or partially autonomous driving mode in accordance with Levels 1 to 4 of the SAE J3016 classification. Here and below, SAE J3016 refers to the respective standard as of June 2018.

[0062] Thus, at least partially automatically guiding the vehicle may include guiding the vehicle according to a fully automatic or fully autonomous driving mode according to Level 5 of the SAE J3016 classification. At least partially automatically guiding the vehicle may also include guiding the vehicle according to a partially automatic or partially autonomous driving mode according to Levels 1-4 of the SAE J3016 classification.

[0063] In some implementations, the electric vehicle guidance system includes a display device and / or a camera system.

[0064] According to some implementations, the readout rate is equal to or less than the refresh rate of the display device, and / or preferably, the readout rate is greater than the frame rate, for example, at least twice the frame rate, for example, at least five times the frame rate.

[0065] Further implementations of the electric vehicle guidance system according to the present invention can be derived directly from the various embodiments of the computer-implemented method according to the present invention, and vice versa. In particular, individual features and corresponding descriptions and advantages related to the various implementations of the computer-implemented method according to the present invention can be transferred analogously to the corresponding implementations of the electric vehicle guidance system according to the present invention. In particular, the electric vehicle guidance system according to the present invention is designed or programmed to execute the computer-implemented method according to the present invention. In particular, the electric vehicle guidance system according to the present invention executes the computer-implemented method according to the present invention.

[0066] According to a further aspect of the present invention, there is provided a computer program product comprising instructions that, when executed by at least one computing device, for example by at least one computing device of an electric vehicle guidance system according to the present invention, cause the at least one computing device to perform a computer-implemented method according to the present invention.

[0067] According to a further aspect of the invention there is provided a computer readable storage medium storing a computer program according to the invention.

[0068] Further features of the present invention are apparent from the claims, the figures, and the description of the figures. Features and combinations of features described above in the description and described below in the description of the figures and / or shown in the figures may be encompassed by the present invention not only in the respective combinations described, but also in other combinations. In particular, embodiments and combinations of features that do not have all the features of the claims initially defined may also be encompassed by the present invention. Furthermore, embodiments and combinations of features that go beyond or deviate from the combinations of features explicitly recited in the recitation of the claims may also be encompassed by the present invention.

[0069] In the following, the present invention will be described in detail with reference to specific exemplary implementations and respective schematic drawings. In the drawings, identical or functionally identical elements may be designated by the same reference numerals. Descriptions of identical or functionally identical elements are not necessarily repeated with respect to different figures. [Brief explanation of the drawings]

[0070] [Figure 1] 1 is a diagram illustrating a vehicle having an exemplary implementation of an electric vehicle guidance system according to the present invention; [Figure 2] FIG. 1 is a diagram schematically illustrating an example of a projection surface. [Figure 3] 10A and 10B are diagrams illustrating further examples of projection surfaces. [Figure 4] FIG. 2 is a diagram illustrating a schematic view of a vehicle and exemplary positions of a virtual observer from above. [Figure 5] 1 is a schematic diagram of a vehicle and exemplary positions of a virtual observer as viewed from the side; [Figure 6] 3A-3C are diagrams illustrating schematic transitions of the projection plane in an exemplary implementation of the computer-implemented method according to the present invention; [Figure 7] 10 is a schematic diagram of a vehicle and further exemplary positions of a virtual observer as viewed from the side; FIG. [Figure 8] 4 is a flow diagram of a further exemplary implementation of a computer-implemented method according to the present invention. [Figure 9] 4 is a flow diagram of a further exemplary implementation of a computer-implemented method according to the present invention.

[0071] 1 schematically illustrates a vehicle 1 having an exemplary implementation of an electric vehicle guidance system 2 according to the present invention. The electric vehicle guidance system 2 includes a computing device 3, which in some implementations may represent more than one computing device. The vehicle 1, and in particular the electric vehicle guidance system 2, includes a camera 4, e.g., a front-facing camera, a rear-facing camera, or a side-facing camera, and a display device 5.

[0072] The camera 4 is configured to generate a video stream depicting the environment of the vehicle 1 and provide it to the computing device 3. The computing device 3 is configured to control the display device 5 to display images in response to the video stream. To this end, the computing device 3 may execute a computer-implemented method for displaying image data in the vehicle 1 according to the present invention.

[0073] The computing device 3 stores a predetermined initial projection surface 9a and a predetermined final projection surface 9b (see Figure 6). Figure 2 illustrates an example of the shape of the projection surface 6a according to a bowl shape, while Figure 3 illustrates another example, where the projection surface 6b consists of adjacent planes.

[0074] The computing device 3 buffers the video stream in a buffer storage, which is updated at a predetermined frame rate. Note that in some implementations, particularly since the computing device 3 may represent more than one computing device, parts of the computing device 3, such as the buffer storage, may also be included by the camera 4. Alternatively, the buffer storage may be part of the computing device 3 located outside the camera 4.

[0075] The computing device 3 reads image data from the buffer storage at a predetermined read rate. Specifically, the computing device 3 reads each image data in an initial read instance, multiple successive intermediate read instances, and a final read instance. The computing device 3 samples the initial projection plane 9a to determine a set of initial sampling points 11a on the initial projection plane 9a and the final projection plane 9b to determine a set of final sampling points 11b on the final projection plane 9b. Here, each final sampling point 11b is assigned to only one of the initial sampling points 11a. For each intermediate read instance, a set of intermediate sampling points 12, 13 is determined, and each intermediate sampling point 12, 13 interpolates between one of the final sampling points 11b and its assigned initial sampling point 11a to determine an intermediate projection plane 10 including the set of intermediate sampling points 12, 13. For each intermediate read instance, the computing device 3 projects the read image data onto the respective intermediate projection plane 10 and controls the display device 5 to display an intermediate image according to the projected image data.

[0076] In some implementations, the computing device 3 may store predetermined initial viewing parameters for the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i (see FIG. 4), including the initial positions of the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i, and predetermined final viewing parameters for the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i, including the final positions of the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i. For each intermediate readout instance, the computing device 3 determines intermediate viewing parameters of the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i, including respective intermediate positions of the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i, the intermediate positions being on a predetermined curve connecting the initial positions to the final positions. For each intermediate readout instance, a respective intermediate image may be obtained by projecting and transforming the image data.

[0077] As depicted in FIG. 4 , virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i, which may also be referred to as virtual cameras, may be located at any position, particularly around vehicle 1, including the position of rear camera 4. The positions of virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i may be, for example, within a distance range of 1 to 3 meters from vehicle 1. As the positions of virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i change, particularly from an initial position through an intermediate position to a final position, this may also be referred to as virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i flying from one position to another.

[0078] 5 shows the vehicle 1 and one of the positions of the virtual observer 7e as well as the corresponding projection plane 8a as viewed from the side. FIG. 7 additionally shows further positions of the virtual observer 7i corresponding to the positions of the camera 4 as well as the respective projection plane 8c. For example, the initial position may be the position of the virtual observer 7i, and the final position may be the position of the virtual observer 7e. Alternatively, the position of the virtual observer 7i may correspond to one of the intermediate positions.

[0079] 8 shows a flow diagram of a further exemplary implementation of a computer-implemented method according to the present invention. In step 800, a view switch request can be generated by a user. In step 810, the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i fly from an initial position through intermediate positions and reach a final intermediate position in step 820. As described above, the initial projection plane 9a is sampled in step 830, and the final projection plane 9b is sampled in step 840. In step 850, the initial projection plane 9a is transformed through intermediate projection planes 10 as described, and reach the final projection plane 9b in step 860. Also, the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i fly from the final intermediate position to a final position in step 860. In optional step 870, the orientation of the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i is adapted. It will be understood that the described sequence is such that whenever the position of the projection plane and / or the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i changes, this corresponds to a readout instance and a corresponding image is displayed by the display device 5.

[0080] 9 shows a high-level flow diagram of a further exemplary implementation of a computer-implemented method according to the present invention, in which the readout rate is at least twice the frame rate. In step 900, the buffer storage is updated. In step 920, image data is read in an initial readout instance. In step 920, the projection plane is set to the initial projection plane 9a, the read image data is projected onto the initial projection plane 9a as described, and the initial image is displayed. In step 930, it is checked whether the buffer storage has been updated again. If so, the updated image data is read in step 900 in a first intermediate readout instance, the projection plane is changed to the first intermediate projection plane 10 in step 910, and the first intermediate image is displayed in step 920. Otherwise, the previously readout image data is acquired in the first intermediate readout instance, the projection plane is changed to the first intermediate projection plane 10 in step 910, and the first intermediate image is displayed in step 920. These steps are repeated until the final image is displayed.

Claims

1. 1. A computer-implemented method for displaying image data in a vehicle (1), comprising: - a predetermined initial projection plane (9a) and a predetermined final projection plane (9b) are provided, a video stream depicting an environment of the vehicle (1) is received from a camera system (4) of the vehicle (1), the video stream being buffered in a buffer storage, the buffer storage being updated at a predetermined frame rate; image data is read from said buffer storage at a predetermined read rate in an initial read instance, a plurality of successive intermediate read instances, and a final read instance; a set of initial sampling points (11a) on said initial projection plane (9a) and final sampling points (11b) on said final projection plane (9b) are determined, each final sampling point (11b) being assigned to exactly one of said initial sampling points (11a); for each of said intermediate read-out instances, a set of intermediate sampling points (12, 13) is determined, each intermediate sampling point (12, 13) interpolating between one of said final sampling points (11b) and said respectively assigned initial sampling point (11a), and an intermediate projection plane (10) comprising said set of intermediate sampling points (12, 13) is determined; - a computer-implemented method, wherein for each of said intermediate read-out instances, said read-out image data is projected onto said respective intermediate projection surface (10) and an intermediate image according to said projected image data is displayed on a display device (5) of said vehicle (1).

2. - predetermined initial viewing parameters of the virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i) comprising an initial position of said virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i) and predetermined final viewing parameters of said virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i) comprising a final position of said virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i) are provided; for each of said intermediate read-out instances, intermediate viewing parameters of said virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i) are determined, comprising intermediate positions of each of said virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), said intermediate positions lying on a predetermined curve connecting said initial position to said final position; for each of said intermediate read-out instances, said read-out image data is projected onto said respective intermediate projection plane (10) and transformed according to said respective intermediate viewing parameters, said respective intermediate image depending on said projected and transformed image data; 10. The computer-implemented method of claim 1.

3. the initial viewing parameters comprise an initial orientation of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), the final viewing parameters comprise a final orientation of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), and for each of the intermediate readout instances, the respective intermediate viewing parameters comprise a respective intermediate orientation of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i); and / or the initial viewing parameters comprise an initial field of view of the virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), the final viewing parameters comprise a final field of view of the virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), and for each of the intermediate readout instances, the respective intermediate viewing parameters comprise a respective intermediate field of view of the virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i).

3. The computer-implemented method of claim 2.

4. - for said initial read-out instance, said read-out image data is projected onto said initial projection surface (9a) and an initial image according to said projected image data is displayed on said display device (5); and / or For said final read instance, said read image data is projected onto said final projection surface (9b) and a final image according to said projected image data is displayed on said display device (5). A computer-implemented method according to any one of claims 1 to 3.

5. For each of the final sampling points (11b), the intermediate sampling points (12, 13) that interpolate between the final sampling point (11b) of all intermediate read instances and the assigned initial sampling point (11a) are on a straight line connecting the final sampling point (11b) and the assigned initial sampling point. A computer-implemented method according to any one of claims 1 to 4.

6. For each of the final sampling points (11b), the respective distances between the intermediate sampling points (12, 13) that interpolate between the final sampling point (11b) and the assigned initial sampling point (11a) are the same for each pair of successive intermediate read instances. A computer-implemented method according to any one of claims 1 to 5.

7. The computer-implemented method of any one of claims 1 to 6, wherein the readout rate is greater than the frame rate.

8. A computer-implemented method according to any one of claims 1 to 7, characterized in that the readout rate is at least twice the frame rate, for example at least five times the frame rate.

9. A computer-implemented method according to any one of claims 1 to 8, characterized in that the readout rate is equal to or less than the refresh rate of the display device (5).

10. said initial projection plane (9a) is obtained by points that satisfy the following equation: [Equation 1] X, Y, and Z indicate the Cartesian coordinates of each point on the initial projection plane (9a), Z corresponds to the height above a predetermined ground plane where the vehicle (1) is located, n is an even integer equal to or greater than 4, and a ｉ , b ｉ , and c is a predetermined real coefficient; and / or said final projection plane (9b) is given by points that satisfy the following equation: [Equation 2] X, Y, and Z indicate the Cartesian coordinates of each point on the final projection plane (9b), Z corresponds to the height above the ground plane, n' is an even integer equal to or greater than 4, and a ｉ ', b ｉ ', and c' are predetermined real coefficients A computer-implemented method according to any one of claims 1 to 9.

11. - the initial projection surface (9a) comprises an initial base portion obtained by an initial portion of a predetermined ground plane on which the vehicle (1) is located, within which the vehicle (1) is located, and an initial raised portion adjacent to the initial base portion at its outer boundary; and / or - said final projection surface (9b) comprises a final base portion obtained by the final part of the contact plane, within which the vehicle (1) is located, and a final raised portion adjoining said final base portion at its outer boundary; A computer-implemented method according to any one of claims 1 to 9.

12. An electric vehicle guidance system (2) for a vehicle (1), comprising at least one computing device (3), said at least one computing device (3) storing a predetermined initial projection plane (9a) and a predetermined final projection plane (9b); - configured to receive a video stream depicting an environment of the vehicle (1) from a camera system (4) of the vehicle (1) and to buffer the video stream in a buffer storage, the buffer storage being updated at a predetermined frame rate; configured to read image data from said buffer storage at a predetermined read rate in an initial read instance, a plurality of successive intermediate read instances, and a final read instance; - configured to determine a set of initial sampling points (11a) on said initial projection plane (9a) and final sampling points (11b) on said final projection plane (9b), each final sampling point (11b) being assigned to one and only one of said initial sampling points (11a); - configured to determine, for each of said intermediate readout instances, a set of intermediate sampling points (12, 13), each intermediate sampling point interpolating between one of said final sampling points (11b) and said respective assigned initial sampling point (11a), and to determine an intermediate projection plane (10) comprising said set of intermediate sampling points (12, 13); an electric vehicle guidance system (2) configured to, for each of said intermediate read-out instances, project said read-out image data onto said respective intermediate projection surface (10) and control a display device (5) of said vehicle (1) to display an intermediate image according to said projected image data;

13. said readout rate is equal to or less than the refresh rate of said display device (5), and / or the readout rate is greater than the frame rate, for example at least twice the frame rate, for example at least five times the frame rate; An electric vehicle guidance system (2) according to claim 12, characterized in that

14. 12. A computer program comprising instructions which, when executed by at least one computing device (3), cause the at least one computing device (3) to perform the computer-implemented method of any one of claims 1 to 11.

15. A computer-readable storage medium storing the computer program of claim 14.

Images