Displaying image data inside a vehicle using a changing projection surface.

Abstract

To display the image data, initial and final projection planes (9a, 9b) are provided, and the video stream is received and buffered in a buffer storage that is updated at a frame rate. The image data is read from the buffer at a read rate in an initial read instance, intermediate read instances, and final read instances. Respective sets of initial and final sampling points (11a, 11b) on the projection planes (9a, 9b) are determined. For each intermediate read instance, intermediate sampling points (12, 13) that interpolate between the final sampling point (11b) and the respective initial sampling point (11a), and intermediate projection planes (10) including the intermediate sampling points (12, 13) are determined. The read image data is projected onto the respective intermediate projection planes (10), and corresponding images are displayed on the display device (5).

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for displaying image data in a vehicle, an electric vehicle guidance system for a vehicle, a computer program, and a computer-readable storage medium.

Background Art

[0002] A camera system of a vehicle, which may include one or more cameras arranged at different positions of the vehicle, for example, a vehicle surround view system, can be used for driver assistance functions or other functions for autonomous or semi-autonomous driving. One camera image, or two or more camera images from different cameras combined into a stitched view, can be projected onto a predetermined projection plane that is part of a two-dimensional manifold in three-dimensional space, such as a bowl shape or the like. Further, this one projection image or multiple projection images can also be shown as virtual cameras and converted according to the respective viewing parameters of the virtual observer so that the images appear as if they were observed by a virtual observer or captured by a virtual camera. The position and / or orientation of the virtual observer can be set or corrected, for example, by the user or automatically by the vehicle.

[0003] In the application note "360° Wrap-Around Video Imaging Technology Ready for Integration with Fujitsu Graphics SoCs" (September 12, 2022, available at URL https: / / www.fujitsu.com / us / imagesgig5 / 360_OmniView_AppNote.pdf), a video imaging technology enabling a 360-degree view of the vehicle's outer perimeter in real time is described.

[0004] German Patent No. 102015105529A1 relates to a method for transforming an image representing the area surrounding a vehicle from the viewpoint of a virtual camera. The image is represented by a transformation from multiple actual images generated using multiple actual cameras of the vehicle.

[0005] U.S. Patent No. 2021 / 0125401A1 relates to a method for representing the environmental area of ​​an automobile in an image, wherein actual images of the environmental area are captured by actual cameras of the automobile, and the image is generated from these actual images. The image is represented from the viewpoint of a virtual camera within the environmental area, and the image is generated as a bowl shape.

[0006] If the position and / or orientation of the virtual observer changes while the video stream is being displayed, the shape of the projection plane may also change. This change can be implemented as a hard switch, i.e., the projection plane changes without a transition from its initial shape to its final shape. However, changes in the projection plane are not necessarily related to changes in the virtual observer's position and / or orientation and may arise from, for example, changes in user settings. [Overview of the project] [Problems that the invention aims to solve]

[0007] In this regard, an object of the present invention is to provide the possibility of achieving a smooth transition from one projection plane to another. [Means for solving the problem]

[0008] This objective is achieved by the subject matter of each independent claim. Further implementations and preferred embodiments are the subject matter of the dependent claims.

[0009] The present invention is based on the idea of ​​sampling an initial projection plane and a final projection plane, and determining each intermediate projection plane for each of a plurality of consecutive intermediate read instances for reading buffer storage that buffers a video stream. Each intermediate projection plane includes intermediate sampling points that interpolate between sampling points on the final projection plane and corresponding sampling points on the initial projection plane.

[0010] According to aspects of the present invention, a computer implementation method for displaying image data in a vehicle, particularly an automobile, is provided. Herein, a predetermined initial projection plane and a predetermined final projection plane are provided. A video stream depicting the vehicle's environment, particularly the exterior environment, is received from the vehicle's camera system, the video stream is buffered in buffer storage, and the buffer storage is updated at a predetermined frame rate. Image data, particularly the updated image data, is read from the buffer storage at a predetermined read rate. Herein, the reads occur in an initial read instance, a plurality of consecutive intermediate read instances following the initial read instance, and a final read instance following the plurality of intermediate read instances. An initial set of sampling points on the initial projection plane and a final sampling point on the final projection plane are determined. Herein, each final sampling point in the set of final sampling points is assigned to only one of the initial sampling points in the set of initial sampling points.

[0011] For each intermediate readout instance, a set of intermediate sampling points is determined, and each intermediate sampling point in the set interpolates between one of the final sampling points and its assigned initial sampling point. For each intermediate readout instance, an intermediate projection plane containing each set of intermediate sampling points is determined. For each intermediate readout instance, the read image data is projected onto the respective intermediate projection plane, and an intermediate image corresponding to the projected image data is displayed on the vehicle's display device.

[0012] Unless otherwise stated, all steps of the computer implementation method may be carried out by at least one computing device, particularly of a vehicle, which may also be referred to as a data processing device. In particular, the at least one computing device includes at least one processing circuit configured or adapted to carry out the steps of the computer implementation method. For this purpose, the at least one computing device may store, for example, a computer program containing instructions, which, when executed by the at least one computing device, cause the at least one computing device to carry out the computer implementation method. It should be understood that the step of displaying each intermediate image on a display device is such that the at least one computing device controls the display device to display each intermediate image. The display is not necessarily part of the at least one computing device. Alternatively, the display may be part of the at least one computing device.

[0013] A computing device can be understood, in particular, as a data processing device that includes processing circuits. Therefore, a computing device can process data in order to perform computing operations. This may include operations to perform indexed access to data structures, such as lookup tables (LUTs).

[0014] In particular, computing equipment may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more system-on-a-chip (SoCs). Computing equipment may also include one or more processors, such as one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). Computing equipment may also include physical or virtual clusters of computers or other components of the above equipment.

[0015] In various embodiments, the computing device includes one or more hardware and / or software interfaces and / or one or more memory units.

[0016] The memory unit may be implemented as volatile data memory such as dynamic random access memory (DRAM) or static random access memory (SRAM), or as non-volatile data memory such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or flash EEPROM, ferroelectric random access memory (FRAM), magnetoresistive random access memory (MRAM), or phase-change random access memory (PCRAM).

[0017] The initial and final projection planes may be provided, for example, by storing them on the memory unit of at least one computing device, in particular by storing the corresponding mathematical descriptions, lookup tables, or other computer-readable datasets.

[0018] A video stream consists of multiple consecutive frames, each containing image data that is sequentially stored in buffer storage. In particular, when each image data of a frame is stored in buffer storage, the image data of the preceding frame may be deleted or overwritten. However, the buffer storage may also be a ring buffer. In either case, the most recent image data stored in buffer storage changes at a predetermined frame rate and is read at a predetermined read rate. Therefore, the reverse read rate defines the time differences between the initial read instance and the first intermediate read instance, between all pairs of consecutive intermediate read instances, and between the last intermediate read instance and the final read instance. The above frames do not necessarily constitute the entire video stream, but may be part of it.

[0019] For example, for an initial read instance, the read image data may be projected onto an initial projection plane, and an initial image corresponding to the projected image data may be displayed on the display device, in particular, before any of the intermediate images are displayed. Alternatively, or in addition, for a final read instance, the read image data may be projected onto a final projection plane, and a final image corresponding to the projected image data may be displayed on the display device, in particular, after all of the intermediate images have been displayed.

[0020] In particular, each projected image data according to an intermediate readout instance can be displayed directly on a display device or further processed before being displayed on a display device. In other words, an intermediate image can be obtained directly from each projected image data or from further processed projected image data. The further processing described above may include, for example, a filtering or processing step to stitch together image data from different cameras, or transforming the image data according to the viewing position and / or orientation or other viewing parameters of a virtual observer.

[0021] A projection plane can be considered, for example, as a two-dimensional manifold in three-dimensional space. For instance, a projection plane can be defined within a Cartesian coordinate system with coordinates X, Y, and Z using a function f, where Z = f(X,Y). However, projection planes are not always explainable using such a function. In particular, in some implementations, projection planes can also be defined in a piecewise manner using different functions for different parts of three-dimensional space. A projection plane is defined, in particular, within a vehicle coordinate system where the vehicle's position and orientation are known, and especially fixed. For example, the reference point for designating a vehicle may be located at the origin of the coordinate system.

[0022] A projection plane can be part of a 2-dimensional manifold. Consequently, different projection planes can correspond to different parts of the same 2-dimensional manifold. Alternatively, different projection planes can similarly correspond to parts of different 2-dimensional manifolds defined, for example, by different functions f(X,Y).

[0023] Intermediate projection planes can be constructed in various ways, based on their respective intermediate sampling points. For example, they can be obtained by their respective polygonal meshes, where the intermediate sampling points represent the corners of the polygons. Intermediate projection planes can also be obtained by their respective spline surfaces, with intermediate sampling points acting as control points. Intermediate projection planes can also be obtained by other surfaces that interpolate between the intermediate sampling points.

[0024] The number of final sampling points may be less than or equal to the number of initial sampling points. The number of intermediate sampling points for a given intermediate read instance is equal to the number of final sampling points. The intermediate projection planes account for a smooth transition from the initial projection plane to the final projection plane, and for each intermediate read instance, a respective intermediate image is displayed. As a result, the sequence resulting from the displayed images accounts for a smooth transition from the initial view to the final view. Thus, a user of a vehicle looking at the display device does not see a sudden change in the displayed image due to a sudden change within the projection plane.

[0025] In particular, all of the initial projection plane, the intermediate projection planes, and the final projection plane may be different from one another. Thus, the displayed image changes for each of the read instances.

[0026] Since a respective intermediate image is displayed for each intermediate read instance, the read rate is particularly equal to or less than the refresh rate of the display device, preferably equal to the refresh rate. It should be noted that the read rate may also be greater than the frame rate. In other words, each read image data may remain unchanged for two or more consecutive intermediate read instances. Nevertheless, the displayed image generally still changes because the intermediate projection planes generally change. However, in other implementations, the read rate may also be equal to or less than the frame rate, which means that the read image data is different for each pair of subsequent intermediate read instances.

[0027] In some implementations, predetermined initial viewing parameters for a virtual observer, including the initial position of the virtual observer, and predetermined final viewing parameters for the virtual observer, including the final position of the virtual observer, are provided and, in particular, stored in the memory unit of at least one computing device. For each intermediate read instance, intermediate viewing parameters for the virtual observer are determined, including each intermediate position of the virtual observer. Here, the intermediate position lies on a predetermined curve connecting the initial position to the final position. For each intermediate read instance, the read image data is projected onto each intermediate projection plane and transformed according to each intermediate viewing parameter. Each intermediate image depends, for example, on the projected and transformed image data, and corresponds to, for example, this.

[0028] In such an implementation, the change in projection plane from the initial projection plane through the intermediate projection plane to the final projection plane is accompanied by a change in the virtual observer's position. Furthermore, the change in the virtual observer's position occurs as a smooth transition. However, the dynamics of the change in projection plane do not necessarily coincide with the dynamics of the change in the virtual observer's position. In other words, the virtual observer's position may remain unchanged for two or more consecutive read instances of the initial, intermediate, and final read instances.

[0029] Apart from the position of the virtual observer, viewing parameters may include, for example, the orientation of the virtual observer. Viewing parameters may also include or define the field of view of the virtual observer, which may also be referred to as a vehicle camera. Viewing parameters may also include virtual mapping parameters that describe the mapping capabilities of the virtual camera.

[0030] The virtual observer can be positioned at any location within the vehicle's environment. Depending on the virtual observer's position and / or further viewing parameters, the size of the projection surface can vary. In particular, the image data displayed on the display device appears as if it were captured by a virtual camera or viewed by a virtual observer, respectively.

[0031] A curve with an intermediate position can be a straight line or any other curve connecting the initial position to the final position. Users viewing the displayed image may get the impression that the observation point is flying along the curve.

[0032] According to some implementations, the initial viewing parameter includes the initial orientation of the virtual observer, and the final viewing parameter includes the final orientation of the virtual observer. For each intermediate read instance, the respective intermediate viewing parameter includes the respective intermediate orientation of the virtual observer.

[0033] The orientation of a virtual observer can be obtained, for example, by three orientation angles, such as Euler angles. Thus, the intermediate orientation can be considered to lie on the respective curves in the parameter space of orientation angles connecting the initial orientation to the final orientation.

[0034] According to some implementations, the initial viewing parameter includes the virtual observer's initial field of view, and the final viewing parameter includes the virtual observer's final field of view. For each intermediate read instance, each intermediate viewing parameter includes each intermediate field of view of the virtual observer.

[0035] The field of view of a virtual observer can be defined, for example, by two field angles, such as a horizontal opening angle and a vertical opening angle. Thus, intermediate fields of view can be considered to lie on the respective curves in the parameter space of field angles connecting the initial field of view to the final field of view.

[0036] According to some implementations, for each final sampling point, the intermediate sampling points that interpolate between the final sampling point and the assigned initial sampling point for all intermediate read instances lie on the straight line connecting the final sampling point and the assigned initial sampling point.

[0037] In this approach, the shortest connection between each final and initial point can be used for interpolation. Therefore, smaller intermediate projection planes may be required for smoother transitions.

[0038] According to some implementations, for each final sampling point, the distance between each intermediate sampling point that complements the final sampling point and the assigned initial sampling point is the same for each pair of consecutive intermediate read instances.

[0039] In particular, the distance between the initial sampling point and the first intermediate sampling point, as well as the distance between the final sampling point and the last intermediate sampling point, are also equal to the distance between consecutive pairs of intermediate sampling points.

[0040] In other words, the transition is performed in a linear manner, which reduces the computational cost required to calculate the midprojection plane.

[0041] According to some implementations, the read rate is greater than the frame rate.

[0042] Consequently, the same image data can be read from buffer storage for at least two consecutive intermediate read instances. Therefore, the transition appears smoother to the observing user.

[0043] Preferably, the read rate is at least twice the frame rate, for example, at least five times the frame rate. For example, the ratio of read rate to frame rate can be at intervals [5,20].

[0044] As a result, for each camera frame, the same buffered image data is read at least twice, or at least five times, or at least a number of times obtained in the above ratio in each consecutive intermediate read instance. Therefore, the transition appears smoother to the observing user.

[0045] In some implementations, the total number of intermediate read instances can be adjusted to the above ratio such that the initial read instance, all intermediate read instances, and final read instance occur within a single frame, and the image data read in each is the same. Thus, the entire transition can occur within a single frame, which makes the transition appear even smoother. However, the total number of intermediate read instances can also be adjusted to the above ratio such that the initial read instance, all intermediate read instances, and final read instance occur within a predetermined maximum number of frames.

[0046] According to some implementations, the initial projection plane is obtained by points that satisfy the following equation.

[0047]

number

[0048] In particular, the initial sampling points satisfy this equation. Here, X, Y, and Z are the Cartesian coordinates of the respective points on the initial projection plane, Z corresponds to the height above a given ground plane on which the vehicle is located, and n is an even integer equal to or greater than 4, and a i , b i , and c are predetermined real coefficients.

[0049] In other words, the initial projection plane is obtained by an n-th degree polynomial in two variables X and Y, preferably n=4. Consequently, the shape of the initial projection plane can be represented as a bowl, which has a relatively flat portion in the area where the vehicle is located and rises relatively steeply as it moves away from the vehicle. In particular, the vehicle is centered at the origin of the coordinate system, and the XY plane corresponds to the ground plane. This represents a suitable approximation for the distance of the depicted object.

[0050] When n=4, the above formula for Z can be written as follows: Z=W[A(XX s ) 4 +B(YY s ) 4 ]-C.

[0051] In some implementations, the same can be said for the final projection plane. In other words, in such implementations, the final projection plane is obtained by points that satisfy the following equation.

[0052]

number

[0053] In particular, the final sampling point satisfies this equation. Here, X, Y, and Z are the Cartesian coordinates of the respective points on the final projection plane, Z corresponds to the height above the ground plane, and n' is an even integer equal to or greater than 4, and a i ', b i ', and c' are predetermined real coefficients. Preferably, n' = n.

[0054] It should be noted that intermediate projection planes, particularly intermediate sampling points of a given intermediate projection plane, do not necessarily satisfy such equations for two-variable polynomials.

[0055] According to some implementations, the initial projection plane is an initial base portion obtained by the initial part of the ground plane on which the vehicle is located, and includes an initial base portion in which the vehicle is located, and an initial raised portion adjacent to the initial base portion at the outer boundary of the initial base portion. Alternatively, or in addition, the final projection plane is a final base portion obtained by the final part of the ground plane, and includes a final base portion in which the vehicle is located, and a final raised portion adjacent to the final base portion at the outer boundary of the final base portion.

[0056] In particular, the initial base portion and / or the final base portion are convex, or in other words, the initial base portion and / or the final base portion or their respective outer boundaries are convex geometric figures, in particular, convex polygons. In contrast to each base portion, the points on each raised portion have a non-zero height above the ground plane.

[0057] In some implementations, structures using the base and raised portions described can also be used for intermediate projection planes.

[0058] In computer implementations, for use cases or situations not explicitly described herein, it may be stipulated that, in accordance with this method, error messages and / or prompts for user feedback may be output, and / or default settings and / or predetermined initial states may be set.

[0059] A further aspect of the present invention provides an electric vehicle guidance system for a vehicle. The electric vehicle guidance system includes at least one computing device, particularly for a vehicle, which stores a predetermined initial projection plane and a predetermined final projection plane, particularly in one or more memory units of the at least one computing device. The at least one computing device is configured to receive a video stream depicting the vehicle's environment from the vehicle's camera system and to buffer the video stream, for example, in buffer storage of the at least one computing device, which is updated at a predetermined frame rate. The at least one computing device is configured to read image data from the buffer storage at a predetermined read rate in an initial read instance, a plurality of consecutive intermediate read instances, and a final read instance.

[0060] At least one computing device is configured to determine an initial set of sampling points on an initial projection plane and a final sampling point on a final projection plane, with each final sampling point assigned to only one of the initial sampling points. At least one computing device is configured to determine a set of intermediate sampling points for each intermediate readout instance, with each intermediate sampling point interpolating between one of the final sampling points and its assigned initial sampling point, and at least one computing device is configured to determine an intermediate projection plane containing the set of intermediate sampling points. At least one computing device is configured to control the vehicle's display device for each intermediate readout instance to project the readout image data onto its respective intermediate projection plane and to display an intermediate image corresponding to the projected image data.

[0061] An electric vehicle guidance system can be understood as an electronic system configured to guide a vehicle in a fully automated or fully autonomous manner, and in particular without requiring manual intervention or control by the driver or user of the vehicle. The vehicle performs all necessary functions, including steering, deceleration, and / or acceleration, as well as monitoring and recording road traffic and automatically responding accordingly. In particular, an electric vehicle guidance system may implement a fully automated or fully autonomous driving mode in accordance with Level 5 of the SAE J3016 classification. An electric vehicle guidance system may also be implemented as an advanced driver-assistance system (ADAS) that assists the driver for partially automated or partially autonomous driving. In particular, an electric vehicle guidance system may implement a partially automated or partially autonomous driving mode in accordance with Levels 1-4 of the SAE J3016 classification. Here and below, SAE J3016 refers to the respective standard as of June 2018.

[0062] Therefore, guiding a vehicle at least partially automatically may include guiding the vehicle in accordance with a fully automatic or fully autonomous driving mode that conforms to Level 5 of the SAE J3016 classification. Guiding a vehicle at least partially automatically may also include guiding the vehicle in accordance with a partially automatic or partially autonomous driving mode that conforms to Levels 1-4 of the SAE J3016 classification.

[0063] In some implementations, the electric vehicle guidance system includes a display device and / or a camera system.

[0064] According to some implementations, the read rate is equal to or less than the refresh rate of the display device, and / or, preferably, the read rate is greater than the frame rate, for example, at least twice the frame rate, for example, at least five times the frame rate.

[0065] Further implementations of the electric vehicle induction system according to the present invention can be directly derived from various embodiments of the computer implementation method according to the present invention, and vice versa. In particular, individual features and corresponding descriptions and advantages related to various implementations of the computer implementation method according to the present invention can be similarly transferred to corresponding implementations of the electric vehicle induction system according to the present invention. In particular, the electric vehicle induction system according to the present invention is designed or programmed to perform the computer implementation method according to the present invention. In particular, the electric vehicle induction system according to the present invention performs the computer implementation method according to the present invention.

[0066] According to a further aspect of the present invention, a computer program product including instructions is provided. When the instructions are executed by at least one computing device, for example, by at least one computing device of the electric vehicle induction system according to the present invention, the instructions cause the at least one computing device to execute a computer implementation method according to the present invention.

[0067] A further aspect of the present invention provides a computer-readable storage medium for storing a computer program according to the present invention.

[0068] Further features of the present invention are evident from the claims, figures, and descriptions of the figures. Features and combinations of features described above in the description, and features and combinations of features described below in the description and / or shown in the figures, may be included by the present invention not only in each of the described combinations but also in other combinations. In particular, embodiments and combinations of features that do not have all of the features of the claims that are initially clearly defined may also be included by the present invention. Furthermore, embodiments and combinations of features that go beyond or deviate from the combinations of features specified in the enumeration of claims may also be included by the present invention.

[0069] In the following, the present invention will be described in detail with reference to specific exemplary implementations and their respective schematic diagrams. In the drawings, identical or functionally identical elements may be indicated by the same reference numerals. Descriptions of identical or functionally identical elements are not necessarily repeated with respect to different diagrams. [Brief explanation of the drawing]

[0070] [Figure 1] This figure schematically shows a vehicle having an exemplary implementation of the electric vehicle induction system according to the present invention. [Figure 2] This is a schematic diagram illustrating an example of a projection plane. [Figure 3] This diagram schematically illustrates further examples of projection planes. [Figure 4] This diagram schematically shows the exemplary positions of the vehicle and virtual observer as viewed from above. [Figure 5] This diagram schematically shows the exemplary positions of the vehicle and virtual observer as viewed from the side. [Figure 6] This figure schematically shows the transition of projection planes in an exemplary implementation of the computer implementation method according to the present invention. [Figure 7] This diagram schematically shows further exemplary positions of the vehicle and virtual observer as viewed from the side. [Figure 8] This is a flowchart illustrating a further exemplary implementation of the computer implementation method according to the present invention. [Figure 9] This is a flowchart illustrating a further exemplary implementation of the computer implementation method according to the present invention.

[0071] Figure 1 schematically shows a vehicle 1 having an exemplary implementation of the electric vehicle guidance system 2 according to the present invention. The electric vehicle guidance system 2 includes a computing device 3, which in some implementations may also represent two or more computing devices. The vehicle 1, in particular the electric vehicle guidance system 2, includes a camera 4, for example, a front camera, a rear camera, or a side camera, and a display device 5.

[0072] Camera 4 is configured to generate a video stream depicting the environment of vehicle 1 and provide it to computing device 3. Computing device 3 is configured to control display device 5 to display images in accordance with the video stream. For this purpose, computing device 3 may perform a computer implementation method for displaying image data within vehicle 1 according to the present invention.

[0073] The computing device 3 stores a predetermined initial projection plane 9a and a predetermined final projection plane 9b (see Figure 6). Figure 2 illustrates an example of the shape of the projection plane 6a corresponding to a bowl shape, while Figure 3 illustrates another example in which the projection plane 6b consists of adjacent planes.

[0074] Computing device 3 buffers the video stream in buffer storage, which is updated at a predetermined frame rate. Note that in some implementations, particularly since computing device 3 may represent two or more computing devices, parts of computing device 3, such as buffer storage, may also be included by camera 4. Alternatively, buffer storage may be part of computing device 3 located outside camera 4.

[0075] The computing device 3 reads image data from buffer storage at a predetermined read rate. Specifically, the computing device 3 reads image data in an initial read instance, a series of consecutive intermediate read instances, and a final read instance. The computing device 3 samples the initial projection plane 9a to determine a set of initial sampling points 11a on the initial projection plane 9a, and the final projection plane 9b to determine a set of final sampling points 11b on the final projection plane 9b. Here, each final sampling point 11b is assigned to only one of the initial sampling points 11a. For each intermediate read instance, a set of intermediate sampling points 12 and 13 is determined, and each intermediate sampling point 12 and 13 interpolates between one of the final sampling points 11b and its respective assigned initial sampling point 11a to determine an intermediate projection plane 10 containing the set of intermediate sampling points 12 and 13. For each intermediate read instance, the computing device 3 projects the image data read for each intermediate projection plane 10 and controls the display device 5 to display an intermediate image according to the projected image data.

[0076] In some implementations, the computing device 3 can store predetermined initial viewing parameters for virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i (see Figure 4), including their initial positions, and predetermined final viewing parameters for virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i, including their final positions. For each intermediate readout instance, computing device 3 determines the intermediate viewing parameters of virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i, including their respective intermediate positions, where the intermediate positions lie on a predetermined curve connecting the initial position to the final position. For each intermediate readout instance, each intermediate image can be obtained from projected and transformed image data.

[0077] As depicted in Figure 4, the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i, which may also be shown as virtual cameras, can be located at any position, particularly around the vehicle 1, including the position of the rear camera 4. The positions of the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i are, for example, within a distance of 1 to 3 meters from the vehicle 1. When the positions of the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i change, particularly from an initial position through an intermediate position to a final position, this can also be shown as the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i flying from one position to another.

[0078] Figure 5 shows the vehicle 1 and one of the virtual observer 7e positions, as well as the corresponding projection plane 8a, as viewed from the side. Figure 7 additionally shows further positions of the virtual observer 7i corresponding to the camera 4 position, as well as their respective projection planes 8c. For example, the initial position may be the position of the virtual observer 7i, and the final position may be the position of the virtual observer 7e. Alternatively, the position of the virtual observer 7i may correspond to one of the intermediate positions.

[0079] Figure 8 shows a flowchart of a further exemplary implementation of the computer implementation method according to the present invention. In step 800, a view switching request may be generated by the user. In step 810, virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i fly from the initial position through intermediate positions and reach the final intermediate position in step 820. As described above, in step 830, the initial projection plane 9a is sampled, and the final projection plane 9b is sampled in step 840. In step 850, the initial projection plane 9a is changed through the intermediate projection plane 10 as described and reaches the final projection plane 9b in step 860. Also, in step 860, virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i fly from the final intermediate position to the final position. In the optional step 870, the orientations of the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i are adapted. It should be understood that whenever the projection plane and / or the positions of the virtual observers 7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, and 7i change, this corresponds to a readout instance, and the corresponding image is displayed by the display device 5.

[0080] Figure 9 shows a high-level flow diagram of a further exemplary implementation of the computer implementation method according to the present invention, in which the read rate is at least twice the frame rate. In step 900, the buffer storage is updated. In step 920, image data is read in the initial read instance. In step 920, the projection plane is set to the initial projection plane 9a, and the read image data is projected onto the initial projection plane 9a as described, and the initial image is displayed. In step 930, it is checked whether the buffer storage has been updated again. If it has been updated, the updated image data is read in step 900 in the first intermediate read instance, the projection plane is changed to the first intermediate projection plane 10 in step 910, and the first intermediate image is displayed in step 920. Otherwise, previously read image data is acquired in the first intermediate read instance, the projection plane is changed to the first intermediate projection plane 10 in step 910, and the first intermediate image is displayed in step 920. These steps are repeated until the final image is displayed.

Claims

1. A method performed by a computer for displaying image data inside a vehicle (1), - A predetermined initial projection plane (9a) and a predetermined final projection plane (9b) are provided. - A video stream depicting the environment of the vehicle (1) is received from the camera system (4) of the vehicle (1), the video stream is buffered in buffer storage, and the buffer storage is updated at a predetermined frame rate. - Image data is read from the buffer storage at a predetermined read rate during the initial read, multiple consecutive intermediate reads, and final read. - A set of initial sampling points (11a) on the initial projection plane (9a) and a final sampling point (11b) on the final projection plane (9b) are determined, and each final sampling point (11b) is assigned to only one of the initial sampling points (11a). - For each of the intermediate readouts, a set of intermediate sampling points (12, 13) is determined, and each intermediate sampling point (12, 13) interpolates between one of the final sampling points (11b) and its respective assigned initial sampling point (11a), thereby determining an intermediate projection plane (10) that includes the set of intermediate sampling points (12, 13). - A method performed by a computer, wherein, for each of the intermediate readouts, the readout image data is projected onto the respective intermediate projection surface (10), and an intermediate image corresponding to the projected image data is displayed on the display device (5) of the vehicle (1).

2. - A predetermined initial viewing parameter for the virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), including the initial position of the virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), and a predetermined final viewing parameter for the virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), including the final position of the virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), are provided. - For each of the intermediate readouts, the intermediate viewing parameters of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i) are determined, including the intermediate position of each virtual observer (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), and the intermediate position lies on a predetermined curve connecting the initial position to the final position. - For each of the intermediate readouts, the readout image data is projected onto the respective intermediate projection plane (10) and transformed according to the respective intermediate viewing parameters, and each intermediate image depends on the projected and transformed image data. A method performed by a computer according to claim 1, characterized in that

3. - The initial viewing parameters include the initial orientation of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), the final viewing parameters include the final orientation of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), and for each of the intermediate readouts, each of the intermediate viewing parameters includes each of the intermediate orientations of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), and / or - The initial viewing parameters include the initial field of view of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), the final viewing parameters include the final field of view of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i), and for each of the intermediate readouts, each of the intermediate viewing parameters includes the respective intermediate fields of view of the virtual observers (7a, 7b, 7c, 7d, 7e, 7f, 7g, 7h, 7i). A method performed by a computer according to claim 2, characterized in that

4. - With respect to the initial readout, the readout image data is projected onto the initial projection surface (9a), and an initial image corresponding to the projected image data is displayed on the display device (5), and / or - Regarding the final readout, the readout image data is projected onto the final projection surface (9b), and the final image corresponding to the projected image data is displayed on the display device (5). A method performed by a computer according to claim 1, characterized in that

5. For each of the final sampling points (11b), the intermediate sampling points (12, 13) that interpolate between the final sampling point (11b) and the assigned initial sampling point (11a) for all intermediate readouts lie on a straight line connecting the final sampling point (11b) and the assigned initial sampling point. A method performed by a computer according to claim 1, characterized in that

6. For each of the final sampling points (11b), the distance between each of the intermediate sampling points (12, 13) that interpolate between the final sampling point (11b) and the assigned initial sampling point (11a) is the same for each pair of consecutive intermediate readouts. A method performed by a computer according to claim 1, characterized in that

7. The method performed by a computer according to claim 1, characterized in that the read rate is greater than the frame rate.

8. The method performed by the computer according to claim 1, characterized in that the read rate is at least twice the frame rate.

9. The method performed by the computer according to claim 1, characterized in that the read rate is equal to or less than the refresh rate of the display device (5).

10. - The initial projection plane (9a) is obtained by a point that satisfies the following equation, [Math 1] X, Y, and Z represent the Cartesian coordinates of the respective points on the initial projection plane (9a), Z corresponds to the height above a predetermined ground plane on which the vehicle (1) is located, n is an even integer equal to or greater than 4, and a ｉ , b ｉ , and c are predetermined real coefficients, and / or - The final projection plane (9b) is obtained by a point that satisfies the following equation, [Math 2] X, Y, and Z represent the Cartesian coordinates of the respective points on the final projection plane (9b), Z corresponds to the height above the ground plane, n' is an even integer equal to or greater than 4, and a ｉ ', b ｉ ', and c' are predetermined real coefficients. A method performed by a computer according to claim 1, characterized in that

11. - The initial projection plane (9a) is an initial base portion obtained by the initial portion of a predetermined ground plane on which the vehicle (1) is located, and includes an initial base portion in which the vehicle (1) is located, and an initial raised portion adjacent to the initial base portion at the outer boundary of the initial base portion, and / or - The final projection plane (9b) is the final base portion obtained by the final part of the ground plane, and includes the final base portion in which the vehicle (1) is located, and the final raised portion adjacent to the final base portion at the outer boundary of the final base portion. A method performed by a computer according to claim 1, characterized in that

12. An electric vehicle guidance system (2) for a vehicle (1), comprising at least one computing device (3), wherein the at least one computing device (3) stores a predetermined initial projection plane (9a) and a predetermined final projection plane (9b), and - The system is configured to receive a video stream depicting the environment of the vehicle (1) from the camera system (4) of the vehicle (1), and to buffer the video stream in buffer storage, the buffer storage being updated at a predetermined frame rate, - The system is configured to read image data from the buffer storage at a predetermined read rate during the initial read, multiple consecutive intermediate reads, and the final read. - Configured to determine a set of initial sampling points (11a) on the initial projection plane (9a) and a final sampling point (11b) on the final projection plane (9b), wherein each final sampling point (11b) is assigned to only one of the initial sampling points (11a). - For each of the intermediate readouts, the system is configured to determine a set of intermediate sampling points (12, 13), each intermediate sampling point is configured to interpolate between one of the final sampling points (11b) and each of the assigned initial sampling points (11a), and to determine an intermediate projection plane (10) that includes the set of intermediate sampling points (12, 13). - An electric vehicle guidance system (2) configured to control the display device (5) of the vehicle (1) to project the read image data onto the respective intermediate projection surface (10) for each of the intermediate readouts, and to display an intermediate image corresponding to the projected image data.

13. - The read rate is equal to or less than the refresh rate of the display device (5), and / or - The read rate is greater than the frame rate. The electric vehicle guidance system (2) according to claim 12, characterized in that

14. A computer program including instructions, wherein, when executed by at least one computing device (3), the instructions cause the at least one computing device (3) to perform the method executed by the computer described in claim 1.

15. A computer-readable storage medium for storing the computer program described in claim 14.

Images