METHOD AND DEVICE FOR REPRESENTING AN IMAGE

Abstract

Method for representing an image (104), comprising the following steps: - representing (201) a first camera image (120) and at least one further camera image (122, 124, 126) on a three-dimensional structure to obtain a composite camera image (140) formed from the camera images (120, 122, 124, 126), transforming (203) the represented camera image (140) into a plane to obtain a planar camera image (144), and transmitting (205) the planar camera image (144) via an interface to a display installation (136) displaying the image (104).

Description

Scope of the invention The present invention relates to a method and a device for representing an image. The invention also relates to a computer program for executing the process and a memory medium containing the program recording. State of the art In Europe, augmented reality (AR) applications are particularly well-known, for example, applications for smartphones or tablets. These applications enhance the real-world scene on a device's display by adding artificial objects. A common application is the display of points of interest (POIs) which can be interesting places for the user in the environment but also for other applications such as the representation of furniture in their apartment or models of construction games presented before unpacking the game. In the automotive field, AR virtual reality applications are also known, both as a video representation of a combined instrument cluster, as a central display, and as a head-up display. These applications are dominated by driving-related information, such as navigation instructions or destination guidance, street names, target objects for the adaptive cruise control (ACC) system, and points of interest (POIs) like parking spaces, accessible parking spaces for people with disabilities, or other commonly used points of interest. Such applications are described, for example, in DE 10 2013 016 244 A1. Purpose of the invention In relation to this context, the invention aims to develop a method for representing an image, as well as a device applying this method and a computer program enabling the development of these techniques. Description and advantages of the invention To this end, the invention relates to a method for re-presenting an image, comprising the following steps of re-presenting a first camera image, and at least one other camera image on a three-dimensional structure to obtain a composite camera image formed from the camera images, transforming the camera image represented in a plane to obtain a flat camera image, and transmitting this flat camera image via an interface to a display installation displaying the image. In other words, the invention makes it possible to combine several camera images to obtain a wide-angle image and to integrate any virtual content into it. The process applies, for example, to a motor vehicle. Camera images are image data provided by the vehicle's cameras, which capture the vehicle's surroundings. The representation step is performed using an appropriate transformation rule that represents the image points of the camera images (images provided by one or more cameras) onto a three-dimensional structure. Thus, it is not necessary to represent the image onto a physically existing three-dimensional structure; rather, the image can be transformed using transformation rules and computational means to obtain a representation comparable to a representation on a real-world dimensional structure. Similarly, one can transform using another transformation rule that represents the image points of the camera image onto a plane.In the supply stage, we will have image data representing the camera's flat image, data which allows the image to be displayed using the display installation. Thus, according to one embodiment, the process transforms into a step of capturing camera images using at least one camera from a vehicle. In the case of multiple cameras, these can have different angles of view. Therefore, the camera capture areas can be directly juxtaposed or overlap to obtain the composite camera image, which combines the images with virtually no "seams." The process may further include a step of displaying the image using the display system. The display system is, for example, the screen in the vehicle. This allows the vehicle driver to have a 360-degree view. In the representation stage, a virtual object can also be represented on the three-dimensional structure. In this way, the image can be supplemented with virtual content. Since the virtual object is represented on the three-dimensional structure, it will be processed using the image data corresponding to the composite camera image, as displayed; thus, the virtual object will be virtually undistorted, integrated into the displayed image using the display system. In the representation stage, camera images can be represented on a sphere, cylinder, or key as a three-dimensional structure. Such structures are suitable for generating an image of the environment. The method includes a step of shifting the camera's flat image within the plane. This results in a shifted camera image. In the delivery step, the shifted camera image is applied to the interface to display the image from a different viewing direction. This allows the image representation to be modified if the viewer desires a different perspective. In one embodiment, the transformation step involves opening the camera image by slicing it along a slicing line to distort it in the plane. This allows for a sort of unspooling of the camera image in the plane. In this case as well, as in the previously described image modification steps, it is preferable to modify the image(s) using a computer or, more generally, computational tools. Thus, in the transformation step, the position of the cutting line can be selected using the viewing direction signal defined by the intended viewing direction of the image. In this way, the image that the display system can represent, corresponding to the user's desired viewing direction, is generated. For example, in the representation stage, four camera images can be displayed on the three-dimensional structure. From these camera images, a 360° representation of the combined, copied camera image is obtained. This is advantageous for applications involving vehicles already equipped with four cameras offering forward, rearward, and side-facing views. The process can, for example, be applied in the form of a program or circuit or a mixed form of programs and circuits in a control device. The invention also relates to a device for applying the steps of the process in suitable installations, specifically designed to apply the process. The device according to the invention allows for a rapid and efficient solution to the problem at hand. The device includes at least one processing unit for handling signals or data, at least one memory unit for storing signals or data, at least one interface with a sensor or actuator for recording sensor signals and transmitting data or control signals to the actuator, and / or at least one communication interface for transmitting or receiving data embedded in communication protocols. The processing unit includes, for example, a signal processor, a microcontroller, or a similar device, and the memory unit is flash memory, EEPROM, or magnetic memory.The communication interface allows data to be received or transmitted via wireless and / or cable links, and this communication interface thus allows data transmitted by cable to be received or transmitted, to be applied by electrical or optical means to data transmission lines and to receive and transmit under such conditions. The device according to the invention is an electrical apparatus that processes sensor signals and, based on these signals, generates control signals and / or data signals. The device includes an interface in the form of a circuit and / or a program. In the case of a circuit implementation, the interface is, for example, part of an ASIC system that performs various functions. It is also possible to have an interface with its own integrated circuit or one that is at least partially composed of discrete components. In the case of a programmatic implementation, the interface is a program module existing within the microcontroller alongside other program modules. The invention also advantageously relates to a computer program product or computer program with program code recorded on a machine-readable medium or memory such as semiconductor memory, hard disk or optical memory to apply, transform and control the steps of the process according to any of the characteristics developed above, in particular when the program product or more simply the program is executed by a computer or calculator. Drawings The present invention will be described below in more detail, with the aid of embodiment examples shown in the accompanying drawings in which: Figure 1 is a diagram of a vehicle equipped with a re- device presentation of an image based on an example of a finished product, Figure 2 is the flowchart of an example implementation of a method of representing an image, Figure 3 is an augmented virtual reality representation on 360° corresponding to an example of implementation, Figure 4 is a representation of the image of Figure 3 on a sphere corresponding to an example of implementation, Figure 5 is an example of how to represent the image of figure 3 on a cylinder, Figure 6 is an example of how to implement a representation according to an opening angle of 90°, Figure 7 is a representation with an opening angle of 150° corresponds to an example of implementation, Figure 8 shows a representation of the image of Figure 3 on a three-dimensional structure of an example implementation under a form already partially developed in a plan Figure 9 represents the image of Figure 8 according to an example of realization in an even more expansive form, Figure 10 is a representation of the image of Figure 9 in a form developed even further in the plane and corresponding to an example of implementation, Figure 11 is a representation of the image of Figure 10 in a form already practically unfolded in the plan according to a example of implementation, Figure 12 shows the image of Figure 11 in an unstretched form. in the plan according to an example of implementation, Figure 13 is a shifted representation of the image of the figure 12, corresponding to an example of implementation, Figure 14 represents virtual objects and a projection surface. video montage in a final situation according to an example of realization- tion, Figure 15 represents the virtual objects of Figure 14 and the over- video projection face according to an example of implementation with pro- jettion, Figure 16 represents the virtual objects of Figure 15 and the over- video projection screen as a transformation of virtual objects Here is an example of what can be done, Figure 17 shows an image with embedded virtual objects. according to an example of implementation, and Figure 18 shows the image of Figure 17 with an as- distortion sphere of the representation corresponding to an example. Description of an embodiment of the invention Figure 1 shows a diagram of a vehicle 100 comprising a device 102 for displaying an image 104 according to one embodiment. The vehicle 100 is, for example, equipped with four cameras 110, 112, 114, 116 to provide a first image from camera 120, a second image from camera 122, a third image from camera 124, and a fourth image 126. According to variations of the embodiment, the device 102 can use a smaller or larger number of images from cameras 120, 122, 124, 126. Device 102 has at least one representation installation 130, a transformation installation 132 and a supply installation 134. According to one embodiment, the device further comprises cameras 110, 112, 114, 116 and, in addition or alternatively, a display installation for displaying the image 104. The display installation 130 includes an interface for receiving images from cameras 120, 122, 124, and 126, and from these images, for providing a composite image 140. To achieve this, the display installation 130 projects the images from cameras 120, 122, 124, and 126 onto a three-dimensional structure. To represent the images 120, 122, 124, and 126 as the image from camera 140, an appropriate representation or transformation rule is applied. For example, the representation installation 130 transforms the camera images 120, 122, 124, 126 into a coordinate system at the basis of a three-dimensional structure, to represent the camera images 120, 122, 124, 126 on the three-dimensional structure.The representation installation 130 represents the images of cameras 120, 122, 124, 126 in areas of the three-dimensional structure which are defined by the images of cameras 120, 122, 124, 126 in the viewing directions of the basic viewing angles. The deformation or transformation installation 132 transforms the camera image 140 represented into a plane to obtain a planar camera image 142. When the three-dimensional structure is considered as a framework, this three-dimensional structure can be deformed with the camera image 140. To deform the camera image 140, represented, an appropriate transformation rule or transformation prescription is applied according to which, to deform the camera image 140 represented, an appropriate representation rule or transformation rule can be used which transposes the image points of the copied camera image 140 onto the three-dimensional structure to retain only the plane. According to one example, the transformation installation 132 cuts the camera image shown 140 along a cutting line in order to somehow unwind the camera image The image 104 is represented by a line 140 starting from the cutting line. By choosing a position for the cutting line, the viewing direction represented by the image 104 can be determined. To define this viewing direction, the transformation device 132 is used, as shown in an example, to record a viewing direction signal 152 defining the viewing direction. This signal is suitable for determining the position of the cutting line representing the image 104 in the desired viewing direction. For example, the viewing direction signal 152 can be provided by an observer of the image 104 using the adjustment device 154 available to the observer. The supply unit 134 transmits the image from the plane camera 140, or the image data 144 representing the image from the plane camera 140, to the interface of the display unit 136. The display unit 136 is configured according to this implementation example to display the image to be represented 104 using the image data 144 to make it visible to an observer. The image 104 contains image information from the original camera images 120, 122, 124, and 126. According to one embodiment, the device 102 comprises an installation 146 to provide a virtual object 148 representing the object data 150. The representation installation 130 also represents the object data 150 on the three-dimensional structure so that the represented camera image 140 will be supplemented by the representation of the virtual object 148. This processing of the represented image 140, already described in installations 132, 134, displays the virtual object 148 as a representation of images integrated into the image 104. Figure 2 shows a flowchart of an example implementation of an image representation method. The method can be applied using the device described in Figure 1. In step 201, camera images (images provided by the cameras) are displayed on a three-dimensional structure. This display generates a composite image from the camera images. In one example, a virtual object is also displayed on the three-dimensional structure. The virtual object can be represented... positioned in a predefined location on the three-dimensional structure. This predefined location can be determined by object recognition on at least one of the camera images or on the presented camera image. In this way, the virtual object can be associated with an object copied from the image data. In step 203, the resulting camera image is transformed into a plane to obtain a flat camera image. In step 205, the flat camera image is fed into an interface for a display system, suitable for displaying the camera image. Optionally, camera images are captured in step 207, for example, using one or more cameras attached to one or more vehicles. Similarly, optionally, the image based on the flat camera image will be displayed in step 209 using the display installation. According to an example implementation, in step 211 the flat camera image is shifted within the plane to obtain a shifted camera image. This shifting modifies the viewing angle of the displayed image. A 360° augmented reality system based on this process in a vehicle provides representations analogous to a 360° video system initially designed for parking maneuvers. This system displays the vehicle from an external perspective, for example, an aerial view, to give the driver a better view. To generate the 360° circular view, four cameras mounted on the vehicle are used. The images from the four cameras are then seamlessly stitched together. Figure 3 shows a 104 image as a 360° augmented reality (AR) representation, based on an example implementation. It could be a 104 image like the one described in Figure 1. The invention enables a vivid, clear, and distortion-free representation of AR 148 content in a 104° to 306° image. This presents a significant challenge compared to camera-based systems. The unique aspects lie in the representation of wide viewing angles (FoV in English terminology), particularly for angles greater than 180°. To achieve this, we propose a suitable transformation that works both for the camera image, for example, using the output images described in Figure 1, and also for all virtual content to obtain a meaningful image 104. The result is a 360° view with virtual content with which one can continue to interact. Viewing angles smaller than 360° can also be applied, and from the user's perspective, these are sometimes more interesting. However, the perceptible difference between a typical scenario and the proposed scenario increases with the viewing angle. Figure 4 shows a representation of the image of Figure 3 on a sphere according to an example embodiment. In the case of the representation, it is the image of camera 140 as described above with reference to Figure 1. In a classic way, four camera images (called above camera images, i.e., images coming from a camera) are represented on a 3D circular structure, for example on a sphere, a cylinder or a key. Figure 4 shows a representation on a three-dimensional sphere-shaped structure. Figure 5 shows the image of Figure 3 on a cylinder, which is another example of this implementation. The representation shown is a camera image 140 like the one described above using Figure 1. This provides a representation of the camera images on a three-dimensional structure in the form of a sphere. Next, the geometry thus formed with a virtual camera will be outlined and displayed. For wider viewing angles, the difficulty lies in the fact that distortion increases with the camera's distance from the center of the virtual camera, and therefore the representation is no longer as accurate. A viewing angle greater than 180° is not geometrically feasible. Figure 6 shows, as an example, a 104 image with an opening angle of 90° according to an example of an implementation. Figure 7 compared to figure 6 shows a 104 image with an opening angle of 150°, also corresponding to an example of implementation. Figure 8 is a representation 840 of the image of Figure 3 on a three-dimensional structure in a form already partially unfolded in a plane and corresponding to another embodiment. Image 840 can thus represent a camera image that is already in an initial intermediate stage towards a planar camera image. Therefore, it cannot be a camera image that has already been partially transformed. Representation 840 shows that camera images are first represented on a sphere or cylinder, as illustrated in Figures 4 and 5; then, the images are transformed by this geometry after representation. In particular, a cylinder can be "unrolled" onto a plane to obtain a regular image. Figure 8 shows that the camera image depicted has been cropped along a cutting line. The resulting ends are already stretched to distort the camera image towards a plane. Figure 9 shows a 940 representation of the image of figure 8 in an even more unfolded form in the plane and which corresponds to an example of realization. Figure 10 is a 1040 representation of the image of figure 9 which has been further developed in the plane and corresponds to an example of realization. Figure 11 shows a representation 1140 of the image of figure 10 in a practically unrolled form, in the plane and which corresponds to an example of realization. Figures 8-11 thus show the step-by-step unfolding of geometry in a plane. Figure 12 represents the image of Figure 11 in a now fully unfolded form 140 and corresponding to a Another embodiment. This could be the planar image 142 as described in figure 1. According to a conventional approach, the virtual camera is rotated to also control the viewing direction of the image from the flat camera 142. After unrolling, this is no longer directly possible because the movement would then be limited to a single flat screen for the field of view. The virtual camera also remains unchanged. Instead, the goal is to shift the image on the unrolled screen. Alternatively, the process can be described as if the cross-section were shifted within the resulting cylinder. Figure 13 shows an offset representation 1342 of the camera plan image of Figure 12 according to another realization example which was displayed by a display installation as described using Figure 1. Figures 12 and 13 thus show an image shift for the geometry. Figure 14 shows the virtual object 150 and a video projection surface 1450 in a new output state according to an example embodiment. The objects 150 can take the form of available object data as described in Figure 1. Figure 15 shows a representation of the virtual objects 150 of Figure 14 and the video image projection surface 1450 in a projection according to an example of implementation. The 1450 video projection surface represents a three-dimensional structure like the one described for figure 1. Figure 16 shows the representation of a transformation of virtual objects according to an example of an implementation. The virtual objects presented above using Figure 15 and the flat video projection surface in a flat camera image 142 is transformed and next to the video projection surface, transformed 1650, the virtual transform presents objects 148. In addition, a virtual camera 1660 is represented. Thus, the representation of figure 16 will be made using a corresponding planar camera image 142 as described using figure 1. The steps of the process for embedding virtual objects into an image are shown in Figures 14-16. The virtual objects 148 in image 142 are incorrectly drawn as if they were direct and without further adaptation. The correct representation of camera images on a sphere or cylinder implicitly guarantees the correct arrangement. In this case, however, this geometry will be modified after representation. Each step is thus transmitted to a virtual content 150. Furthermore, it should be noted that the position of the virtual content 148 in image 142 always depends on the position of the virtual camera 1660. To eliminate this effect, the virtual contents 150 of the other transformation are projected onto the surface of the sphere / cylinder. The transformation in the plane is performed exclusively by the objects located on this surface, as shown in Figures 14 to 16. Une formulation correspondante de la transformation en pseudo-Code se présente comme suit en langage de programmation qui ne peut se traduire : / / linput] shape parameters, range [0,1] float factor2Cy]; float factor2Plane; / / [input] longitude shift, to realize rotation of viewing direction float longitudeShift / / linput, output] point to transform vec3 p; / / calculate distance and use for depth float lengthXY = length(p.xy}); / / project to video canvas float longitude = atan(p.y, p.x) + longitudeShift; float latitude = pi*0.5 —- atan(p.z / lengthXY); / / calculate canvas shape p.x = cos(longitude)*sin(latitude)*(1.0 - factor2Cyl) + cos(longitude)* factor2Cyl*(1.0 - factor2Plane) + factor2Plane; p.y = sin(longitude)*sin(latitude)*(1.0 - factor2Cyl) + sin(longitude)*factor2Cyl*(1.0-factor2Plane) + (longi- study) / pi*factor2Plane; pz = cos(latitude)*(1.0 - factor2Cy]) + (pi*O0.5 - latitude)*factor2Cyl; The pseudo-code, through its two shape parameters, also allows for the creation of any intermediate shape between a classic sphere and an unrolled surface. The value of factor2Cyl defines the weighting between the sphere and the cylinder (sphere = 0, cylinder = 1), while the value of factor2Plane represents the weighting relative to the plane (closed geometry = 0, fully unrolled geometry = 1). All intermediate values ​​are valid. Choosing only 0 and / or 1 for both values ​​significantly simplifies the second half of the pseudo-code. Figures 8 to 11 show images and different parameters for each case. The final result, as shown in Figure 3, is obtained by applying the transformation to the video image and the virtual 3D objects represented. This representation can be further distorted, analogous to an aspherical rearview mirror, according to the user's wishes, as shown in Figures 17 and 18. Figure 17 shows an image 104 in which virtual objects 148 have been embedded according to an example of implementation. Figure 18 shows image 104 of figure 17 with an aspheric distortion of the representation corresponding to an example.

Claims

VJ Method for representing an image (104), comprising the following steps: representing (201) a first camera image (120) and at least one other camera image (122, 124, 126) on a three-dimensional structure to obtain a composite camera image (140) formed from the camera images (120, 122, 124, 126), “the representation step. (201) further forming on the three-dimensional structure, object data (ISO) representing a virtual object (148} so that the image of the ramera (140) is completed with a representation of the virtual object (148), transform (203) the represented camera image (140) in a plane to obtain a planar camera image (1.44), “camera image (.142) being opened along a 15 section line in a plane and transmitting (205) the planar camera image (144) to an interface for a display installation (136) displaying the image (104). 2*) Method according to claim 1s 3û characterized in that it comprises a step (201) of capturing camera images (120, 122, 124, 126) using at least one camera (1WS 112, 114, 116) of a vehicle (100) and / or a step (209) of displaying the image (104) using a display installation (136). 3a) Method according to one of claims 1 to 2, characterized in that in the representation step. (201) the camera images (120, 122, 124, 126) are represented on a sphere, a cylinder or a key as a 3Q three-dimensional structure. 4”) Method according to one of the preceding claims, characterized in that it comprises a step (211) of shifting the plane image of the camera (144) in the plane to obtain an image shifted by 35 camera (1342), - in the transmission step (20S) the shifted camera image (1342) is provided to the interface to display the image (104) according to a modified viewing direction. s 5*) Method according to claim 4, characterized in that in the transformation step (203) a position of the cutting line is selected using a view direction signal (152) for the view direction for Fanning (104), 10 h®) Method according to the preceding claims, characterized in that in the representation step (201) four camera images (120, 122, 124, 126) are represented on the three-dimensional structure in order to obtain the composite camera image (142) in the form of a representation on 36CF from the camera images (120, 122, 124, 126). 7®) Device (102) comprising units for carrying out the steps of the method according to any one of claims 1 to 6, 2$ characterized in that it comprises at least one camera (110, 112, 114, 1X6) and / or a display installation as well as a representation installation (130), a transformation installation (132) and a supply installation (134). 25 8®) Computer program comprising instructions for program allowing the execution of the steps of the method according to one of claims 1 to 6 when this program is executed on a computer.

Images