Method for providing a 360-degree image of an environment of a vehicle

Abstract

The invention relates to a method for providing a 360-degree image (23) describing an environment (10) of a vehicle (1). The method comprises: generating (S1) an equirectangular image (20) of the environment (10) by applying an image generation algorithm (21) to several camera images (22), each of which describes a section (11, 12, 13) of the environment (10) of the vehicle (1), by a control device (2) of the vehicle (1); receiving (S2) the generated equirectangular image (20) by a mobile device (3); generating (S3) the 360-degree image (23) of the environment (10) by applying an image conversion algorithm (24) to the received equirectangular image (20) by a control apparatus (4) of the mobile device (3); and providing (S4) the generated 360-degree image (23) by the mobile device (3).

Description

[0001] Method for providing a 360-degree image of an environment of a vehicle

[0002] The invention relates to a method for providing a 360-degree image of an environment of a vehicle. The invention also relates to a control device for a vehicle, a control apparatus for a mobile device and a computer program product which are configured to carry out at least individual steps of such a method.

[0003] A vehicle may comprise at least one camera that is configured to capture at least a part of the environment of the vehicle. If, for example, cameras are arranged in a front area, a rear area and in side areas of the vehicle, the environment of the vehicle may be completely captured at least in a horizontal plane. Such an image of the environment may be of interest to a user of the vehicle, for example, if he or she wishes to view the environment of the vehicle using a mobile device, for example as part of a multimedia experience, to check a parking position and / or for remote monitoring of the vehicle. It is therefore reasonable to be able to provide images of the environment of the vehicle as quickly and reliably as possible for the user's mobile device.

[0004] US 2018 / 0075652 A1 discloses a method for generating a virtual reality image. The method comprises receiving a panoramic image obtained by synthesizing images taken with a camera in several directions from a specific reference point in a space.

[0005] US 10,740,972 B2 discloses a method for displaying and controlling a virtual vehicle view with an environment representation. The representation of the environment may be supplemented by iconography.

[0006] US 10,277,813 B1 discloses a viewing device, such as a virtual reality headset, that allows a user to view a panoramic scene captured by one or more video capture devices, which may include several cameras that simultaneously capture 360-degree video data.

[0007] It is the object of the invention to provide a solution by which images of an environment of a vehicle may be provided quickly and reliably for a mobile device.

[0008] The object is solved by the subject matter of the independent patent claims.

[0009] A first aspect of the invention relates to a method for providing a 360-degree image. The 360-degree image describes an environment of a vehicle. The invention is based at least on the realization that, in connection with providing 360-degree images for mobile devices, it is often problematic to transmit the 360-degree images, which typically have a large amount of data, to the mobile device in real time or at least close to real time and thus sufficiently quickly. Therefore, according to the invention, measures are to be taken to at least improve the speed of transmission.

[0010] The method comprises generating an equirectangular image of the environment. The equirectangular image is generated by applying an image generation algorithm to several camera images, each of which describes a section of the environment of the vehicle. This means that exactly one equirectangular image may be generated from the several camera images. The generation of the equirectangular image is performed by a control device of the vehicle. The control device of the vehicle may, for example, be integrated into at least one camera of the vehicle and / or be spatially separated from it. The control device is, for example, an electronic control unit (ECU). The image generation algorithm comprises at least one criterion and / or rule, during the execution of which the equirectangular image of the environment is generated, this means created or generated, from the several camera images.

[0011] In a preferred example, it may be provided that the several camera images together describe the entire environment of the vehicle, at least in a horizontal plane. The several camera images may then be combined to form a 360-degree view of the environment. The equirectangular image therefore shows the 360-degree environment of the vehicle. However, the individual camera images and thus also the equirectangular image may be limited in a vertical direction, for example depending on a detection range of the respective camera, which captures the respective camera image of the several camera images and provides it for the method.

[0012] In other words, the individual camera images from several cameras are fused by applying the image generation algorithm so that they describe a 360-degree representation of the environment of the vehicle. However, this 360-degree representation is generated by converting the environment captured as a spherical object into a flat object, this means projecting it as a flat object. The generated equirectangular image is therefore a flat image and not a representation on a virtual sphere. This may result in a distorted representation of at least one part of at least one of the camera images, at least locally in the equirectangular image. For example, a certain partial image of one of the camera images may be enlarged, reduced or distorted in the equirectangular image. Applying the image generation algorithm may generate or perform an equirectangular projection of the several camera images. By fusing the several camera images and the described projection, a reduction in the amount of data may be achieved, since, for example, the several camera images alone have a larger amount of data than the generated equirectangular image.

[0013] The generation of the equirectangular image may be preceded by the several camera images being captured by several cameras of the vehicle and transmitted to the control device. The several camera images may therefore be provided to the control device in advance.

[0014] The method comprises receiving the generated equirectangular image by a mobile device. The mobile device may be assigned to a user, in particular a driver of the vehicle. The mobile device is, for example, a tablet and / or a smartphone. To receive the equirectangular image, the mobile device comprises, for example, a communication interface. The vehicle may also have a communication interface so that the communication interface of the vehicle may transmit the generated equirectangular image directly or indirectly to the communication interface of the mobile device via a communication connection. The mobile device then receives the transmitted equirectangular image.

[0015] The communication connection is, for example, cable-free, in particular wireless. For example, it may be configured as a wireless local area network (WLAN for Wireless Local Area Network), a Bluetooth connection and / or a mobile data network, for example based on the Long Term Evolution (LTE), Long Term Evolution Advanced (LTE-A), Fifth Generation (5G) or Sixth Generation (6G) mobile radio standard.

[0016] Before receiving the generated equirectangular image by the mobile device, for example, a request from the mobile device to the vehicle may have been made. For example, the vehicle may only transmit the equirectangular image to the mobile device after receiving such a request from the mobile device. Alternatively or additionally, reception may only take place if the mobile device has, for example, an application or another function that requests the equirectangular image, in particular automatically. Alternatively or additionally, the vehicle, in particular a function of the vehicle, may trigger and thus initiate the transmission of the equirectangular image to the mobile device.

[0017] The method comprises generating the 360-degree image of the environment by applying an image conversion algorithm to the received equirectangular image. The 360-degree image is generated by a control apparatus of the mobile device. The control apparatus of the mobile device is a control unit which is at least configured to convert the received equirectangular image into the 360-degree image. The image conversion algorithm comprises at least one criterion and / or rule, during the execution of which the 360-degree image of the environment is generated from the equirectangular image by converting the equirectangular image into the 360-degree image. The image conversion algorithm may be based on known methods for converting an equirectangular image into a 360-degree image and / or generally into a non-equirectangular image. The 360-degree image is not an equirectangular image. The 360-degree image differs at least in part from the equirectangular image. The 360-degree image may alternatively be referred to as a panoramic image or 360-degree panoramic view image.

[0018] The method includes providing the generated 360-degree image using the mobile device. The 360-degree image is now made available for further use by the mobile device. For example, it may be further processed, displayed, manipulated and / or transmitted to another mobile device by the mobile device.

[0019] In a preferred example, the procedure is carried out repeatedly. Not only is a single equirectangular image generated and received, but numerous equirectangular images are generated and received one after the other. This means that not only a static 360-degree image may be provided, but a dynamic or moving 360-degree image, in particular a video comprising individual 360-degree images.

[0020] The method enables the mobile device to receive 360-degree images of the environment of the vehicle generated on the basis of the several camera images in real time or with a comparatively short time delay, this means close to real time. By converting into the equirectangular image and transmitting it, the several camera images are pre-processed in such a way that the amount of data to be transmitted is reduced, enabling fast and thus at least near-real-time transmission. This improves the transmission of images from the vehicle to the mobile device. Overall, images of an environment of a vehicle are therefore provided quickly and reliably to a mobile device.

[0021] An embodiment provides for a fusion matrix to be applied to the several camera images when the image generation algorithm is applied. The fusion matrix describes at least one pixel in the equirectangular image for each pixel of the respective camera image. If, for example, an image from a front camera of the vehicle is assumed as the camera image, for each individual pixel of this camera image from the front camera it is determined where this pixel is to be positioned in the equirectangular image. This information is included in the fusion matrix, so that by applying the fusion matrix to the front camera image and the other camera images, it is known where the respective pixel of the respective camera image is to be positioned or displayed in the generated equirectangular image. Analogous to the camera image for the front camera, the assignment of the individual pixels in the equirectangular image is also known for the other camera images of the several camera images and is recorded in the fusion matrix or is included in the fusion matrix.

[0022] It may be the case that exactly one pixel in the rectangular image is provided for each pixel in the camera images. Alternatively or additionally, it is possible that, for example, a pixel of the respective camera image is to be distributed over several pixels that lie next to each other in the equirectangular image. This may be caused, for example, by the fact that a local resolution in the camera image is different from the resolution in the fused image of the environment, this means in the equirectangular image.

[0023] The fusion matrix therefore describes the assignment of the pixels in individual camera images to the pixels in the equirectangular image for all several camera images together. The several camera images are therefore fused by applying the fusion matrix to the several camera images as part of applying the image generation algorithm. The fusion matrix may be in the form of a list, table and / or matrix, for example. The fusion matrix illustrates how the equirectangular image may be reliably generated.

[0024] A further embodiment comprises that the fusion matrix for a respective pixel of the respective camera image not only describes the at least one pixel in the equirectangular image, but also a color value. The color value describes a color of the at least one pixel in the equirectangular image, in particular relative to a color of the pixel in the camera image. If, for example, a first pixel is viewed in the camera image of the front camera and two pixels are assigned to it in the equirectangular image, for example, a color value of a first pixel of the two pixels in the equirectangular image may have a different color than a second pixel of the two pixels in the equirectangular image. For example, if the at least one pixel in the camera image has a first color value, which describes for example the color white, the first pixel and the second pixel in the equirectangular image each have only a portion of the first color value. For example, the first pixel may appear dark gray and the second pixel light gray. The color value may therefore be specified, for example, as a percentage or factor that is related to the color value of the pixel in the camera image. The color value may, for example, comprise a red component, a green component and / or a blue component. Instead of the color value, a gray value may be considered, for example, if the camera images are gray value images and not color images. When generating the equirectangular image, not only are the pixel positions of the individual pixels of the camera images taken into account, but also changes in the color that may occur when generating the equirectangular image. For this reason, the fusion matrix is particularly suitable for generating the equirectangular image, as it considers both the described local pixel assignment and special features in the color and thus in the color value.

[0025] In addition, an embodiment comprises generating for each individual pixel in the respective camera image a pixel image to determine the fusion matrix. The pixel image describes only the individual pixel in a first state and all other pixels in a different second state. The pixel image may be understood as a binary image. For example, the pixel image is completely black except for the individual pixel, which is displayed in white or another color other than black. For example, if there are only the states 0 and 1 , the individual pixel is in state 0 and all other pixels are in state 1 or vice versa. Alternatively or additionally, the pixel image may have two different states measured in gray values and / or color values.

[0026] It is particularly relevant that exactly one pixel image is generated for each individual pixel in the respective camera image. For example, if the camera image has ten pixels, ten pixel images are generated for exactly this camera image, whereby the pixel in the first state is located at a different position in the pixel image compared to the other pixel images. In this example, ten different pixel images are generated.

[0027] A fusion matrix determination algorithm is applied to the respective generated pixel image, whereby an entry for the respective pixel in the first state is determined for the fusion matrix. In the above example, the fusion matrix determination algorithm is applied to the ten pixel images one after the other or simultaneously. The fusion matrix determination algorithm may therefore be applied consecutively or simultaneously to all pixel images of the several camera images, whereby in total all pixels of the several camera images are considered. If the application is carried out one after the other, an entry for the fusion matrix may be generated for exactly one pixel by each individual application, namely the individual pixel in the first state. The entry describes the position of the pixel in the equirectangular image, whereby this position may comprise one pixel or several pixels, and / or the color value of the pixel in the equirectangular image. The fusion matrix determination algorithm comprises, for example, at least one criterion and / or rule, during the execution of which the fusion matrix is determined. The fusion matrix is preferably determined once, in particular once for the vehicle. The determined fusion matrix may then be applied to camera images from the several cameras for which it was determined. When applying the image generation algorithm, the fusion matrix determination algorithm is therefore not applied again. This illustrates exactly how the fusion matrix may be generated. The generated fusion matrix may be stored in a storage unit of the vehicle, for example.

[0028] Furthermore, an embodiment comprises that while the fusion matrix determination algorithm is applied to the respective generated pixel image, a standard image is assumed for the other camera image in the case of two camera images and / or the other camera images in the case of more than two camera images. In the standard image, all pixels are in the second state. If, for example, the second state specifies that the respective pixel is displayed in black, all camera images are completely black except for the camera image currently being considered, for which the pixel image is generated and considered. Data is therefore assumed for all camera images of the several camera images when determining the fusion matrix, wherein the data is either the pixel image or the standard image. When determining the fusion matrix, for example, exactly one pixel image is always considered together with at least one standard image, in particular several standard images.

[0029] In the case of four camera images, for example, the fusion matrix determination algorithm is applied successively to the pixel images of all four camera images, whereby a single pixel image together with three standard images for the other three camera images are always used. In the example with ten pixels per camera image, the fusion matrix determination algorithm is first applied to ten pixel images with standard images kept constant for the other three camera images. Then the camera image for which the pixel images are provided changes, for example, to another camera image of the four camera images, so that the standard image is now considered for the camera image for which the pixel image was previously considered. In this example, a total of forty different pixel images are used to be able to determine all entries for the fusion matrix using the fusion matrix determination algorithm. This generates a particularly reliable or accurate fusion matrix. Furthermore, all pixels of the several camera images may be considered. This makes it possible to recognize overlapping image areas, as pixels in the equirectangular image, for example, are then assigned several times. A preferred embodiment comprises that the fusion matrix determination algorithm considers at least one arrangement of several cameras, which capture the several camera images, in or on the vehicle. For example, positions and / or orientations of the cameras are determined or predefined and are considered by the fusion matrix determination algorithm. The positions and / or orientations may, for example, be specified in relation to a vehicle coordinate system. Alternatively or additionally, it may be provided that the fusion matrix determination algorithm considers not only the arrangements of the several cameras, but also optical properties of the respective cameras. For example, a design of the lens of the respective camera may be taken into account. In a preferred example, the several cameras have fisheye lenses that have a field of view of 180 degrees, in particular 190 degrees. Other configurations of cameras are possible. The information on the arrangements of the cameras at least helps to ensure that the fusion matrix may be determined particularly accurately.

[0030] In an additional embodiment, at least one overlapping area is detected when the image generation algorithm is applied. The overlapping area is described by at least two of the several camera images. For example, there may be lateral areas of the vehicle that are arranged laterally to a central axis of the vehicle, which lie in the fields of view or detection ranges of several cameras. For example, the field of view or detection range of the front camera may at least partially overlap with a field of view or detection range of the respective side camera of the vehicle. This may also be the case for the rear camera and the side cameras. There are therefore sections of the environment that are described by at least two of the camera images. These are therefore described at least twice or more.

[0031] The overlapping areas are recognized and are only displayed once in the generated equirectangular image. If, for example, images from cameras with an aperture angle of 190 degrees are used four times to capture the four camera images, in total images covering 760 degrees are available. However, only 360 degrees are required to describe the environment. The sections described by the camera images are therefore reduced by more than half by identifying the overlapping areas and displaying them only once. This illustrates how the size of the equirectangular image is significantly reduced compared to the raw data, meaning the several camera images. The smaller equirectangular image generated is therefore particularly suitable for the desired transmission to the mobile device.

[0032] In another embodiment, it is intended that the image generation algorithm is applied to exactly four camera images. In particular, these camera images are captured by the front camera, the rear camera and two side cameras of the vehicle. The respective side camera is arranged, for example, in a respective side mirror of the vehicle. The front camera and the rear camera are located in particular in a lower area of the vehicle body in the vertical direction of the vehicle, for example in or on a bumper, in particular on or near a license plate and / or a manufacturer's logo of the vehicle. These cameras may be used to capture the entire environment of the vehicle.

[0033] It may be possible, for example, to apply a correction algorithm to the several camera images before generating the equirectangular image, for example to homogenize or adjust the brightness of the several camera images. This is useful if, for example, due to the position of the sun or the position of another light source relative to the vehicle, at least one of the several camera images is more exposed and therefore has a higher brightness than at least one of the other camera images. In this case, methods for homogenizing differently illuminated or differently bright camera images may be used in order to standardize the lighting conditions in the several camera images. This means that unwanted differences in brightness are no longer displayed in the 360-degree image or are at least reduced.

[0034] Furthermore, an embodiment provides for several mobile devices to receive the generated equirectangular image independently of one another. It is therefore not envisaged that, for example, only a single mobile device of exactly one user of the vehicle may receive the equirectangular image, but that it may be received simultaneously by several mobile devices. In particular, the mobile devices may be of different types and / or be assigned to different users or other persons. Several users or persons may therefore benefit simultaneously from the fact that the 360-degree image is provided to them, whereby this is based on the several transmissions of the equirectangular image. The individual mobile devices are independent of each other, for example. They therefore do not require any data from each other, but generate the 360-degree image individually. This is particularly advantageous if the 360-degree image is to be made available to numerous different users individually.

[0035] An additional embodiment comprises that when the image conversion algorithm is applied, the received equirectangular image is projected onto a virtual sphere in order to generate the 360-degree image. This means that not only is the equirectangular image displayed directly, but measures are taken to generate a three-dimensional image from it, in which, for example, distortions due to the conversion to the equirectangular image are corrected again and thus reversed, so that, for example, distortions that are expected when projecting onto a virtual sphere are generated. The projection onto the virtual sphere may be intended to emphasize the character of a more spatial and three-dimensional image compared to the flat equirectangular image.

[0036] An embodiment comprises that providing the 360-degree image comprises a display of the 360-degree image by at least one display device of the mobile device. The mobile device may, for example, comprises a screen, in particular a touch screen, on which the 360-degree image is displayed. Providing therefore comprises not only that the mobile device may access the 360-degree image, but also that the 360-degree image is actually displayed or shown. Alternatively or additionally, the display device may be a different type of screen, such as a computer screen, a television screen and / or a tablet screen.

[0037] A model of the vehicle may be integrated into the 360-degree image. The model may initially be positioned in the center, for example. It is then surrounded by the environment according to the 360-degree image. The model may be used, for example, to clarify a vehicle position in the 360-degree image. It enables an outside perspective of the vehicle and its environment by displaying the environment and the vehicle. There are numerous display options for how the content of the 360-degree image may be processed and displayed for the user.

[0038] It may be provided that the 360-degree image is also displayed by a display device in the vehicle, in particular the 360-degree image that is displayed by the mobile device. The control device may comprise a further image generation algorithm that is applied to the several camera images to generate the 360-degree image in the vehicle.

[0039] In addition, an embodiment provides for the mobile device to comprise at least one operating element by which a perspective on the environment and / or a scaling of at least a part of the environment in the displayed 360-degree image is adjustable. For example, in the case of the touch screen, a perspective on the environment may be changed using at least one finger of the user's hand. For example, the user may point their finger at any point in the displayed 360-degree environment and the vehicle and the rest of the environment are then displayed in such a way that it is assumed that an observer is positioned at the point. This allows numerous different views (perspectives) of the environment and / or the vehicle in the environment to be adjusted. To scale at least a part of the environment, an operating gesture may be performed with two fingers, for example. In this case, the two fingers may be moved away from or towards each other in order to reduce or enlarge the displayed environment. This may be used, for example, to bring individual parts of the displayed image and therefore the environment into focus. Alternative or additional operating gestures or actions that go beyond the two examples described are possible.

[0040] The described changes or influences on the 360-degree image only occur locally on the respective mobile device. For example, they do not affect the other mobile devices that also receive the equirectangular image. The operation carried out with the operating element is recorded and evaluated locally on the mobile device without the need for or the occurring of any transmission or other data exchange with the vehicle. This means that several users may interact with the 360-degree image at the same time, for example using several mobile devices, each in their own way and each view the environment and / or the vehicle from different perspectives and / or at different scales. This enables a multi-user system comprising of several mobile devices and the vehicle without mutual interference, which is particularly convenient.

[0041] A further aspect of the invention relates to a control device for a vehicle. The control device is configured to carry out the steps of the method described above intended for the vehicle, in particular for the control device of the vehicle. The control device performs these steps.

[0042] A further aspect of the invention relates to a control apparatus for a mobile device. The control apparatus is configured to carry out the steps of the method described above intended for the mobile device. The control apparatus carries out these steps.

[0043] The control device and the control apparatus may, for example, comprise a processor device. This may comprise at least one microprocessor, microcontroller, FPGA (Field Programmable Gate Array) and / or DSP (Digital Signal Processor). Furthermore, it may comprise program code, which may alternatively be referred to as a computer program product. The program code may be stored in a data memory of the processor device.

[0044] One aspect of the invention relates to a vehicle with the control device and a mobile device with the control apparatus. The vehicle is, for example, a motor vehicle, such as a passenger car, a truck, a bus, a motorcycle and / or a moped.

[0045] A further aspect of the invention relates to a system or arrangement comprising the vehicle and the mobile device. The system is configured to perform the method described above. The system performs the method. Another aspect of the invention relates to a computer program product. The computer program product is a computer program. The computer program product comprises instructions which, when the program is executed by several computers, such as by the control devices and / or the control apparatus, cause the computers to perform the steps of the method.

[0046] The embodiments described in connection with the method according to the invention, both individually and in combination with one another, apply accordingly, insofar as applicable, to the control device according to the invention, the control apparatus according to the invention and the computer program product according to the invention. The invention comprises combinations of the described embodiments.

[0047] The figures show:

[0048] Fig. 1 a schematic representation of a vehicle with several cameras,

[0049] Fig. 2 a schematic representation of a signal flow graph of a method for providing a 360-degree image, and

[0050] Fig. 3 a schematic representation of the creation of a fusion matrix.

[0051] In the figures, identical components are marked with the same reference signs.

[0052] Fig. 1 shows a vehicle 1 with a control device 2. Outside the vehicle 1 there is a mobile device 3, which is here a smartphone. Alternatively or additionally, the mobile device 3 may be a tablet and / or another electronic device. The mobile device 3 has a control apparatus 4. In the example outlined here, the mobile device 3 is assigned to a user 5. The user 5 may be, for example, a driver and / or another occupant of the vehicle 1 and / or another person. The user 5 has already left the vehicle 1 here. Alternatively, the user 5 may be in the vehicle 1 .

[0053] The vehicle 1 comprises several cameras. Here, for example, it comprises a front camera 6, two side cameras 7 and a rear camera 8. In the example shown, the side cameras 7 are arranged in the side mirrors of the vehicle 1 . It is possible that the respective camera has a fisheye lens, meaning a detection range or a field of view of up to 190 degrees. The vehicle 1 may comprise a communication interface 9. The mobile device 3 may also comprise the communication interface 9 (not shown here). At least one piece of information and / or data may be transmitted from the communication interface 9 of the vehicle 1 to the mobile device 3, which is or are received by the communication interface

[0054] 9 of the mobile device 3. A cable-free, in particular wireless, communication connection may exist or be established between the two communication interfaces 9.

[0055] The several cameras of the vehicle 1 are configured to capture an environment 10 of the vehicle 1 . Each individual camera captures at least one section 11 , 12, 13 of the environment 10. The front camera 6 captures the section 11 in a front area of the vehicle 1 . The side cameras 7 capture the sections 12 in side areas of the vehicle 1 and the rear camera 8 captures the section 13 in a rear area of the vehicle 1 . The sections 11 , 12 and the sections 12, 13 partially overlap, for example in overlapping areas 14. The cameras may therefore at least partially capture the same sections 11 , 12, 13.

[0056] Fig. 2 shows steps of a method for providing a 360-degree image 23 of the environment

[0057] 10 of the vehicle 1 . The method is carried out partly by the vehicle 1 and partly by the mobile device 3. In a step S1 , an equirectangular image 20 of the environment 10 is generated. For this purpose, an image generation algorithm 21 is applied to several camera images 22. The several camera images 22 each describe one of the sections 11 , 12, 13 of the environment 10 of the vehicle 1 . The step S1 is carried out by the control device 2 of the vehicle 1 .

[0058] In a preferred example, the image generation algorithm 21 is applied to exactly four camera images 22, which are captured by the front camera 6, the rear camera 8 and the two side cameras 7. It may further be provided that, prior to generating the equirectangular image 20, a correction algorithm for homogenizing different brightness and / or illumination conditions in the individual camera images 22 is applied to the four camera images 22.

[0059] When applying the image generation algorithm 21 , at least one of the overlapping areas 14 described by at least two camera images 22 may be recognized as such and displayed only once in the generated equirectangular image 20. This may reduce an amount of data of the equirectangular image 20 compared to the several camera images 22. The method comprises transmitting the generated equirectangular image 20 from the vehicle 1 to the mobile device 3. In a step S2, the mobile device 3 receives the generated equirectangular image 20 from the vehicle 1 , for example via the communication connection. Receiving may be performed, for example, by the communication interface 9 of the mobile device 3.

[0060] In a step S3, a 360-degree image 23 of the environment 10 is generated. For this purpose, an image conversion algorithm 24 is applied to the received equirectangular image 20. The step S3 is carried out by the control apparatus 4 of the mobile device 3 and thus locally on the mobile device 3. When applying the image conversion algorithm 24, the received equirectangular image 20 may, for example, be projected onto a virtual sphere, whereby the 360-degree image 23 is generated.

[0061] In a step S4, the generated 360-degree image 23 is provided by the mobile device 3. It may, for example, be displayed by a display device 25 of the mobile device 3. For example, the vehicle 1 itself may be displayed in the displayed 360-degree image 23 in the form of a virtual vehicle representation 26. The virtual vehicle representation 26 is a model of the vehicle 1 . Here, for example, the virtual vehicle representation 26 is positioned centrally in the environment 10.

[0062] It may be provided that the mobile device 3 comprises at least one operating element by which a perspective on the environment 10 and / or a scaling of at least part of the environment 10 in the displayed 360-degree image 23 is adjustable. For example, a touch gesture or touch action on a touch screen as the display device 25 is suitable for this purpose.

[0063] Overall, the step S1 is performed by the vehicle 1 and the steps S2 to S4 are performed by the mobile device 3. It may be that several mobile devices 3 receive the generated equirectangular image 20 independently of one another, in particular simultaneously or at the same time.

[0064] Fig. 3 illustrates the determination of a fusion matrix 30. The fusion matrix 30 is applied, for example, when the image generation algorithm 21 is applied to the several camera images 22. The fusion matrix 30 describes at least one pixel in the equirectangular image 20 for each pixel of the respective camera image 22. It may be provided that the fusion matrix 30 describes not only a single pixel in the equirectangular image 20 for the respective pixel of the respective camera image 22, but several pixels, which in particular lie next to each other. The fusion matrix 30 may also describe a color value for the respective pixel of the respective camera image 22 in the equirectangular image 20. The color value describes a color of the at least one pixel in the equirectangular image 20, in particular relative to a color of the pixel in the camera image 22.

[0065] To determine the fusion matrix 30, a pixel image 32 may be generated for each individual pixel in the respective camera image 22. This pixel image 32 may describe exactly one single pixel in a first state 33 and all other pixels in a second state 34 that is different from the first state. For example, the single pixel in the first state 33 may be represented in white and all other pixels in black. The pixel image 32 may alternatively be understood as a binary image that distinguishes exactly the two states 33, 34.

[0066] By applying a fusion matrix determination algorithm 31 individually to the respective generated pixel image 32, an entry for the respective pixel in the first state 33 is determined in the fusion matrix 30. While this is performed for the respective pixel image 32, a standard image 35 describing all pixels in the second state 34 is assumed for each of the other camera images 22. For example, the pixel image 32 may represent the camera image 22 of the front camera 6. Standard images 35 may then be assumed for the rear camera 8 and the two side cameras 7. Depending on how many pixels the respective camera image 22 has, an equal number of different pixel images 32 are generated and viewed. For example, if each camera image 22 has exactly ten pixels, ten pixel images 32 are generated so that the individual pixel in the first state 33 moves to all ten pixel positions of the camera image 22. This is done analogously for the other camera images 22, so that in the example described, an entry for the fusion matrix 30 is determined a total of forty times.

[0067] The fusion matrix determination algorithm 31 may consider at least an arrangement of the several cameras that captured the several camera images 22 in or on the vehicle 1 . The arrangement comprises, for example, a position and / or orientation of the respective camera in relation to, for example, a vehicle coordinate system. Furthermore, optical properties of the respective camera may be considered as well.

[0068] For example, the fusion matrix 30 shown in Fig. 3 is only determined once for the vehicle 1 and its cameras. The determined fusion matrix 30 may then be repeatedly applied to camera images 22 as part of the image generation algorithm 21 and does not have to be determined again and again. Overall, the examples show an equirectangular system for a 360-degree panoramic view for a vehicle 1 with four cameras. In the 360-degree view, the view may only be moved with a single input at a time because the view is generated in the vehicle 1 , which means that several mobile devices 3 may not move freely when one of them controls the view. The solution is to create the equirectangular image 20 from the four camera images 22 of the four environment cameras (cameras) of the vehicle 1 and transmit it to the streaming devices (mobile devices 3), where the equirectangular image 20 is applied to a virtual sphere to create a surrounding view (360-degree image 23) that may be freely navigated with touch.

[0069] The equirectangular system of the 360-degree view provides a unique way of controlling the view on the streaming device (mobile device 3). This means that the user 5 may control his or her own view without affecting the view displayed in the vehicle 1 and / or the view of the other mobile devices 3 connected to the vehicle 1 .

[0070] Furthermore, it may be problematic that the 360 panoramic view generated by the vehicle 1 may not be controlled by an external device (mobile device 3). The solution is to use an external device, meaning the mobile device 3, which receives the transmission of the 360- degree view generated by the cameras from the vehicle 1 and acts as a controller for the free movement of the 360-degree view (360-degree image 23), which provides the user 5 with full control by using the screen of the mobile device 3 to change the viewing angle and zoom in and out.

[0071] The invention provides a way to control the surrounding view on the display device 25 of the vehicle 1 using the mobile device 3 such as a smartphone or tablet, for example with 4G / 5G connection. The users 5 may see around the vehicle 1 and the driver in the vehicle 1 may monitor what the users 5 see.

Claims

Claims1 . Method for providing a 360-degree image (23) of an environment (10) of a vehicle (1), comprising:- generating (S1 ) an equirectangular image (20) of the environment (10) by applying an image generation algorithm (21) to several camera images (22), each of which describes a section (11 , 12, 13) of the environment (10) of the vehicle (1 ), by a control device (2) of the vehicle (1 );- receiving (S2) the generated equirectangular image (20) by a mobile device (3);- generating (S3) the 360-degree image (23) of the environment (10) by applying an image conversion algorithm (24) to the received equirectangular image (20) by a control apparatus (4) of the mobile device (3);- providing (S4) the generated 360-degree image (23) by the mobile device (3).

2. Method according to claim 1 , wherein when applying the image generation algorithm (21 ) a fusion matrix (30) is applied to the several camera images (22), which describes for each pixel of the respective camera image (22) at least one pixel in the equirectangular image (20).

3. Method according to claim 2, wherein the fusion matrix (30) for the respective pixel of the respective camera image (22) not only describes the at least one pixel in the equirectangular image (20), but also describes a color value that describes a color of the at least one pixel in the equirectangular image (20), in particular relative to a color of the pixel in the camera image (22).

4. Method according to one of claims 2 or 3, wherein for determining the fusion matrix (30) for each individual pixel in the respective camera image (22) a pixel image (32) is generated which describes only the individual pixel in a first state (33) and all other pixels in a second state (34) different therefrom, and an entry in the fusion matrix (30) for the respective pixel in the first state (33) is determined by applying a fusion matrix determination algorithm (31) to the respective generated pixel image (32).

5. Method according to claim 4, wherein while the fusion matrix determination algorithm (31) is applied to the respective generated pixel image (32), a standard image (35) describing all pixels in the second state (34) is assumed for the other camera image (22) and / or the other camera images (22) of the several camera images (22).

6. Method according to claim 4 or 5, wherein the fusion matrix determination algorithm (31 ) considers at least one arrangement of several cameras capturing the several camera images (22) on the vehicle (1).

7. Method according to any one of the preceding claims, wherein, when applying the image generation algorithm (21), at least one overlapping region (14) described by at least two of the several camera images (22) is recognized and represented only once in the generated equirectangular image (20).

8. Method according to any one of the preceding claims, wherein the image generation algorithm (21) is applied to exactly four camera images (22), which are captured in particular by a front camera (6), a rear camera (8) and two side cameras (7) of the vehicle (1).

9. Method according to any one of the preceding claims, wherein several mobile devices (3) receive the generated equirectangular image (20) independently of each other.

10. Method according to any one of the preceding claims, wherein upon applying the image conversion algorithm (24), the received equirectangular image (20) is projected onto a virtual sphere to generate the 360-degree image (23).11 . Method according to any one of the preceding claims, wherein providing the generated 360-degree image (23) comprises displaying the 360-degree image (23) by at least one display device (25) of the mobile device (3).

12. Method according to claim 11 , wherein the mobile device (3) comprises at least one operating element, by which a perspective on the environment (10) and / or a scalingof at least a part of the environment (10) is adjustable in the displayed 360-degree image (23).

13. Control device (2) for a vehicle (1 ), which is configured to carry out the steps of a method according to any one of claims 1 to 8 provided for the vehicle (1 ).

14. Control apparatus (4) for a mobile device (3), which is configured to carry out the steps of a method according to any one of claims 1 and 9 to 12 provided for the mobile device (3).

15. Computer program product comprising instructions which, when the program is executed by several computers, in particular the control device (2) according to claim 13 and the control apparatus (4) according to claim 14, cause the same to perform a method according to any one of claims 1 to 12.

Images