PRODUCTION OF PANORAMIC IMAGES

Description

[0001] The present invention relates to the production of panoramic images, in particular to the production of panoramic images with reduced artifacts.

[0002] Panoramic images cover a wide viewing angle, which does not necessarily encompass a 360° field of view. They are usually created by stitching together multiple images taken from different perspectives.

[0003] In one method of creating a panoramic image, several individual images are captured sequentially. Between each image, the camera is panned slightly. The individual or partial images are then stitched together to form a panoramic image. Errors can occur during the stitching process. Errors caused by the entrance pupil not aligning with the camera's axis of rotation can be minimized through careful design. However, problems arising from the movement of objects within the scene being filmed are not easily remedied.

[0004] According to another method, combining individual images into a single image can be achieved not using time-division multiplexing, but rather channel-division multiplexing. In this method, the individual images are simultaneously captured through multiple channels with different viewing directions and then stitched together. Due to the adjacent arrangement of the channels, parallax problems arise during stitching. In both cases, therefore, stitching problems occur due to inconsistencies between the individual images: once due to the different acquisition times and the other time due to the spacing between the optical channels.

[0005] US Patent 9,838,687 discloses a panoramic camera setup with four cameras with overlapping fields of view. The cameras transmit their video signals to a server over a network. The server decodes the frames received from the cameras and stitches them together to create panoramic images. The server includes a module that generates a suggested field of view in the panoramic video and is configured to transmit video signals to client devices.

[0006] The object of the present invention is to produce panoramic images that have improved quality, for example due to reduced inconsistency between the individual partial shots.

[0007] This task is solved by the subject matter of the independent patent claims. Further embodiments are defined in the dependent patent claims.

[0008] The present application is based on the idea that two types of individual images are captured, resulting in a set of simultaneously captured images – a first set of images – and a further set of sequentially captured images – a second set of images. For this purpose, simultaneous images – the first set – are generated using multiple optical channels of a camera system. At least one further image – called the second set of images – generated using one or more of these channels, is added at a time offset from the first set of images and / or from each other. The set of time-shifted images can therefore be taken from the same "location" or with a small baseline, so that they exhibit no or only negligible parallax.The other set of simultaneously acquired images does not suffer from motion inconsistencies between the shots. Due to their complementary advantages, these two sets of individual images can be stitched together to create an improved panoramic image. In combination, this can result in a panoramic image that is more artifact-free than one created with only a single set of images, regardless of the overall scene—whether it contains movement or not. Depending on the existing camera system design, the additional engineering effort may be minimal or even nonexistent.

[0009] The device determines a first difference between a first pair of individual images, which cover at least a section of the panoramic field of view and overlap each other in a first overlap area within the panoramic field of view, and determines a second difference between a second pair of individual images from the second set of individual images, which cover the same section of the panoramic field of view and overlap each other in a second overlap area within the panoramic field of view, in the second overlap area. The first pair is selected for constructing the panoramic image if the first difference is smaller than the second difference, and the second pair is selected for constructing the panoramic image in the section if the first difference is larger than the second difference. The first and second overlap areas may coincide.

[0010] In one embodiment, the device is set up to carry out the determination of the first and second difference and the selection of the first or second pair based thereon along a second direction perpendicular to a first direction along which the fields of view covered by the pairs of individual images lie side by side, such that the determination is divided into cross-sections / strips, so that the selection can be different for different cross-sections that run perpendicular to the second direction.

[0011] In one embodiment, the recording of at least one second individual image is carried out such that the individual images of the second set of individual images are recorded with different viewing directions, and a variation in the viewing direction is carried out between the individual images of the second set of individual images, which is brought about by rotating an optical channel of the camera system or by rotating the entire camera system and / or switching a switchable optical element that causes a deflection of a beam path of one optical channel or of a beam path of the several channels of the camera system, and / or rotating a mirror or prism to deflect a beam path of one of the several optical channels or of a beam path of the several channels of the camera system.

[0012] In a further embodiment, the system comprises a second camera system and is configured for the simultaneous acquisition of a third set of individual images, which together cover the panoramic field of view, with several optical channels of the second camera system, wherein the second camera system is arranged offset relative to the first camera system, for the acquisition of at least one fourth individual image, with one of the several optical channels of the second camera system, such that the at least one fourth individual image together (i.e., the at least one fourth individual image taken on its own) or together with a third individual image acquired by one of the several optical channels yields a fourth set of individual images.resulting in images that are taken at different times and also cover the panoramic field of view, and for constructing another panoramic image or a depth map for the panoramic image based on the third set of third individual images and at least one fourth individual image.

[0013] Exemplary embodiments of the present invention are explained in more detail below with reference to the figures, including: Fig. 1 shows a schematic representation of a device for generating panoramic images according to an embodiment; Fig. 2 shows a schematic representation of a camera system according to an embodiment; Fig. 3 shows a schematic representation of a camera system according to an embodiment in which the object to be photographed moves; Fig. 4 is a schematic representation of overlapping pairs of individual images relating to a common overlap area for the analysis of the same with regard to the pair to be preferred for panoramic image generation; and Fig. 5 is a schematic representation of overlapping pairs of individual images relating to a common overlap area for the analysis of the same with regard to the pair to be preferred for panoramic image generation, wherein the analysis is refined section by section.

[0014] Before describing exemplary embodiments using the figures, let us briefly illustrate and motivate the advantages of the exemplary embodiments described below.

[0015] As mentioned in the introductory description, there are various reasons why it can be advantageous to combine multiple images with a relatively small field of view into a single image with a larger field of view. For example, the overall resolution of the image may be higher, the dimensions of the camera system may be smaller, the system's form factor may be more favorable, or it may be possible to design it in a particularly flat configuration.

[0016] There are essentially two methods available for capturing individual images: 1. Time-delayed acquisition of individual images using a single optical channel. For this, the viewing direction is changed between exposures, for example by rotating the entire camera or by deflecting the optical path with a movable mirror. All images are taken from a common viewpoint. 2. Simultaneous acquisition of individual images using multiple optical channels with different fixed viewing directions. In this case, the optical channels are necessarily not located at the same point but are laterally offset from each other. A beam splitter would allow multiple optical channels to share a common nodal point, but this can lead to undesirable light loss.

[0017] In the first method, the individual images differ in their capture time, while in the second method, they differ in the camera's position. This can lead to complications when stitching the individual images together to create the panorama. Artifact-free stitching is only possible for scenes with the following characteristics: 1. Static scenes, meaning there is no movement of the objects in the scene between recording times. 2. All objects in the scene are at the same distance from the camera system or at a distance that is large compared to the offset of the optical channels, by a factor of 100 or more, depending on the angular resolution of the camera system.

[0018] Reconstructing other scenes can also be attempted, but requires increased algorithmic effort. If the reconstruction fails, this manifests as ghost images or misaligned edges and image details.

[0019] Artifacts can be reduced, for example, by quickly switching the viewing direction or by using a small distance between the optical channels.

[0020] The following exemplary implementations take advantage of the individual benefits and scene-specific suitability of the two variants by capturing two sets of individual images: one captured simultaneously and one captured at different times. This allows for an overall improvement in the construction of the panorama image based on the two sets, regardless of the type of scene, i.e., scenes with movement and / or strong variations in depth within the overlapping areas of the individual images.

[0021] Fig. 1 Figure 1 shows a schematic representation of an embodiment of a device 100 for generating a panoramic image. The device 100 comprises an arrangement of several optical channels 110a-110d of a camera system 110. In the present example, the camera system comprises four optical channels; however, the present invention is not limited to four optical channels but can have two or more. The camera system 110 can comprise one or more lenses or one or more other optical elements 115 per channel 110a-110c, which serve to image a respective partial area or partial field of view 150a-150d belonging to the respective channel from a total panoramic field of view 150 onto a respective image sensor or respective image sensor section of an image sensor 111. The channels can run substantially parallel to each other from the image sensor 111 to a deflecting device, such as a mirror, which directs the beam paths of the channels orIt redirects their fields of vision. It can be swivelled between a first, in . Fig. 1 The image shows two positions, one with a total field of view of 150°, and a second position with a total field of view of 150°, which is shifted relative to the 150° in the first position such that the partial fields of view 150d, 150b of channels 110d,b in the second position 150°' essentially overlap with the partial fields of view 150c,a of channels 110c,a in the first position. For this purpose, the partial fields of view 150a-d in each position are arranged here in an overlapping 2x2 array to cover the total field of view, so that the total fields of view 150, 150° in the two positions are offset from each other by one partial field of view in a direction parallel to the beam paths in front of the deflecting device 140 and perpendicular to the pivot axis of the deflecting device 140. Channels 110d and 110b are thus able to cover the entire field of view from the first position 150 when considering the sequence of positions 150 and 150'.The division into a 2x2 coverage area and the number of partial fields of view are not critical, and the number of positions could also be increased. The positions also play a role in capturing a panoramic image, as described below.

[0022] The camera system 110, or its image sensor(s) 111, is / are connected to a decision device 120. This device selects a set of individual images from a set of images acquired according to a specific procedure, which will be described in more detail later. These images are then assembled into a panoramic image. The selection of the set of images can also vary, for example, in a direction perpendicular to the mutual partial field of view offset, as will be described below. The panorama image generator 130 performs the assembly / construction of the panoramic image based on the selected images. Furthermore, in some embodiments, the device can include a mechanism 140 that changes the "viewing direction" of the camera system. As described, the entire field of view of the panoramic camera can be panned to create a panoramic image.This panning, which could be performed manually by the user, can of course also be performed in exemplary embodiments by a device that changes the orientation of the camera system or its overall field of view orientation during a panoramic shooting operation. This could be an alternative to... Fig. 1 For example, this can be achieved by rotating the entire camera system 110, but here it is accomplished by rotating the deflecting device, such as a mirror or prism, so that the beam path of the optical channels of the camera system is deflected, as is shown by the double arrow on the deflecting device 140 in Fig. 1 As indicated... Alternatively, the change in viewing direction or position could be achieved by switching a switchable optical element, whereby the optical element causes a deflection of the beam path. As mentioned, the variation of the camera system's viewing direction could also be done manually by the camera user.

[0023] It should be emphasized that the presence of a deflection device 140 may serve a purpose that arises from a requirement other than changing the position, as used here below for the purpose of panoramic image creation, such as image stabilization. In such a case, an existing camera system can be extended without major design modifications to perform panoramic image creation in accordance with the present application. In the case of the dual use of the deflection device 140 for image stabilization in the direction transverse to the pivot axis, there is also an advantage that the change in viewing direction can be carried out very quickly. That is, the deflection device can execute the position changes rapidly, so that there is only a small time difference between the images taken in the different directions or positions 150, 150'.

[0024] in Fig. 2 The diagram shows a schematic representation intended to illustrate how different sets of individual images could be captured in an exemplary implementation. Fig. 2 For illustrative purposes, the number of positions and also the number of channels in the direction of the relative offset of the overall field of view between the positions is increased to three compared to two, as shown in Fig. 1 was shown, although the division into two adjacent partial fields of view, which are perpendicular to the direction of the overall field of view offset between the positions in Fig. 2 The sequence of shots, as described below, could be automatically executed by the camera system 110, for example, controlled by a control unit of the system 110, such as upon activation of a recording button by the user, possibly in a specific panorama shooting mode.

[0025] The camera system 110 shown in this example therefore has the following features: Fig. 2 via three optical channels A, B, and C, but a different number of optical channels is also conceivable. The optical channels can belong to a single camera, i.e., the channels can be permanently installed in a housing, as is the case in Fig. 1 As illustrated, or alternatively, there could be three individual cameras arranged side by side, at least one of which—in this case, at least the one forming channel A—is swiveling to view the partial fields of view of the other two channels A and C. As can be seen in the figure, the three channels A, B, and C are arranged side by side so that they can simultaneously capture individual images in the direction of a panoramic field of view. The respective viewing directions of the individual optical channels differ slightly from one another, so that the captured individual images have overlapping areas 220, 230. This occurs, for example, when taking a picture in position 1, resulting in images 1A, 1B, and 1C that cover the entire field of view with mutual overlap between 1A and 1B and between 1B and 1C.

[0026] Channel A can be aligned in position 2 so that it faces the same direction as Channel B in position 1, for example by swiveling it to the left. Ideally, in Fig. 2 Assuming that the swiveling of channel A between positions occurs in such a way that its nodal point remains constant, but this is only an example, and a swiveling according to Fig. 1 Alternatively, another beam deflection, as described there, would also be possible and would only slightly alter the concept and nature of the following description of panoramic image generation. In position 2, channel A can generate an image 2A that covers essentially the same partial field of view as image 1B generated by channel B in position 1. Furthermore, position 3 is illustrated, in which the field of view of channel A is aligned with the field of view of channel C in position 1 to generate an image 3A of this field of view.

[0027] To explain the advantages and functionality of the subsequent processing of images 1A-1C, 2A and 3B into a complete field of view, see in Fig. 2 Letters A and B serve as examples of objects in the scene. They are located at different distances within the scene, as illustrated by their different sizes. These objects are used to explain the influence of different shooting times between positions 1, 2, and 3, and the influence of scene depth (i.e., the varying depth or distance of objects within the scene). The explanation also covers how to prevent or reduce the negative impact of these influences on a panoramic image generated from shots 1A-1C, 2A, and 3A. For this purpose, letter B is positioned at a different, greater distance from camera system 110 than letter A.

[0028] Furthermore, it shows Fig. 2 The circled areas 220 and 230 show where the recording areas of cameras A and B, and A and C, respectively, overlap in position 1. This means, for example, in the case of overlap 220, that single image 1A overlaps with single image 1B. More precisely, single image 1A overlaps with the right area of ​​single image 1B in its left area. This means that the same image content is included in this area. However, this image content may differ due to parallax effects, so the image content may not be identical. Similarly, there is overlap area 230 between single images 1A and 1C, as well as (not explicitly shown) overlap areas between the single image pairs 1A+2A and 1A+3A due to the co-alignment of 1B and 1A, and 3A and 1C, respectively. The significance of the overlap areas will be discussed in connection with Fig. 4 discussed in more detail.

[0029] In the example of the Fig. 2 Channel A of camera system 110 captures image 1A, while simultaneously image 1B is captured by channel B and image 1C by camera C. This occurs in position 1. Since the cameras, or rather the optical channels A, B, and C, are not located in the same place but are slightly offset from each other, the captured images can exhibit parallax. This is indicated by the different positions of the previously mentioned letters A and B, which are located within the panoramic field of view.

[0030] In image 1A, the letter B appears to be positioned centrally behind the letter A. Optical channel B, which captures the portion of the visual field to the left of channel A in the viewing direction and produces image 1B, sees the letter B shifted laterally to the left relative to the letter A, since its nodal point, or entrance pupil, is located to the left of that of channel A. It sees the scene slightly from the left. Similarly, optical channel C captures image 1C, in which the letter B appears shifted slightly to the right relative to the letter A. This effect, the parallax effect, occurs when the optical channels are laterally offset from each other and the scene has depth, meaning the objects within it are arranged at different distances.

[0031] It should be emphasized that the order in which positions 1, 2, and 3 are traversed is not relevant and could be changed, but now let us consider position 2, in which optical channel A is rotated to the left to capture single image 2A, and position 3, in which optical channel A is rotated to the right to capture single image 3A. With regard to the partial field of view of the capture, they correspond to single images 1B and 1C, respectively, but unlike single images 1B and 1C, images 2A and 3A are captured from the same perspective as single image 1B. It is accordingly in Fig. 2 It is evident that, since the position of optical channel A does not change, no parallax effect occurs here; that is, the letter B overlaps with the letter A in images 2A and 3A in the same way as in image 1A. Therefore, no parallax occurs in this case.

[0032] In general, it would therefore be advantageous to create a panoramic image based on the set of shots 1A, 2A and 3A, since the arrangement of the letters A and B relative to each other does not differ and thus no artifacts occur.

[0033] However, it is also clear that the set of recordings 1A, 1B and 1C is recorded simultaneously, whereas recordings 1A, 2A and 3A form a set of recordings that must be produced with a time offset from each other, since between the individual recordings the camera or the optical channel A must be panned to the left or right or some other manipulation / switching of the viewing direction must take place.

[0034] If in the example of the Fig. 2 If the arrangement of letters A and B in the panoramic field of view is static, it would not be necessary, for example, to use the simultaneously captured images 2A and 3A. Instead, it would be more advantageous to use the set of sequentially captured images 1A, 1B, and 1C to create an artifact-free panoramic image. However, if the objects in the panoramic field of view are not static but are moving, or if it is a dynamic scene, the situation may be different, as shown in... Fig. 3 shown.

[0035] Fig. 3 schematically shows a similar arrangement to the one in Fig. 2 shown, except that in this case the letter A is moving; in this specific example, this letter performs a clockwise rotation.

[0036] The three individual images 1A, 1B, and 1C, which were captured simultaneously, each show the letter A with the same orientation, since the images were taken at the same time. The subsequent individual images 2A and 3A, captured later through optical channel A after optical channel A was panned left and right, show the letter A slightly rotated. Thus, individual image 2A shows that the letter A is rotated slightly to the right relative to its orientation in images 1A, 1B, and 1C, because individual image 2A was captured after images 1A, 1B, and 1C. Since individual image 3A was captured even later, the letter A in individual image 3A is rotated even further clockwise. Fig. 3 This serves to illustrate the effect that moving objects can present themselves to the observer in different positions in temporally successive individual images, and it should be emphasized once again that the order of passing through positions 1, 2 and 3 could also be changed.

[0037] If the panoramic image were now created from the set of individual images 1A, 2A, and 3A, there would be a noticeable optical artifact due to the successive rotation of the letter A. In this case, it would be more advantageous to create the panoramic image based on the set of individual images 1A, 1B, or 1C.

[0038] Depending on the scene, different scene situations may occur at the overlap areas 220 and 230, i.e., movement and / or scene depth, so that a different set of individual images for panorama image generation might be the most favorable, such as generation based on the set of individual images 1A, 2A or 3A, or the set of individual images 1A, 2A and 1C.

[0039] In order to decide which combination of individual images is more advantageous for generating the panoramic image, it is possible in exemplary embodiments to take a closer look at the aforementioned overlap areas 220, 230 of the individual images, or to examine pairs of individual images that overlap in these overlap areas.

[0040] As in Fig. 2 and 3As shown, image 1A overlaps with image 1B on the left side. The same applies to the pairing of image 1A with image 2A. Similarly, image 1A overlaps with images 1C and 3A on the right side of image 1A.

[0041] Fig. 4 Figure 1 shows a schematic representation intended to further explain the concept of deciding on the composition of the set of images as the basis for panoramic image generation, based on an analysis of the overlapping areas as performed by decision-maker 120. Individual images 1A, 1B, and 2A are shown. Fig. 2 , with the left area of ​​the Fig. 4 The individual images 1A and 1B are shown side by side; in the right area, individual images 1A and 2A are shown.

[0042] In Fig. 4 The overlap areas Ü1 and Ü2 are marked for the individual images. These are the areas that refer to the same part of the panoramic field of view in individual images 1A and 2A. It is obvious that overlap area Ü1 carries the same image information as overlap area Ü2. This is not surprising, since the panoramic image area of ​​the Fig. 2 There is no movement, and therefore the time-shifted individual images 1A and 2A are very similar in the overlap area, since the parallax effect is absent in this case because they were taken from the same perspective, i.e., the location of the canal was always the same. The individual images 1A and 1B are in Fig. 4 The images are also compared side-by-side. The overlap areas Ü1 and Ü3 are marked among these individual images. Here, the similarity in the overlap area is not as pronounced compared to the situation in the image pair 1A and 2A. While the image information about the left half of the letter A is very similar in both overlap areas, due to the parallax effect, the left half of the letter B is more visible in image 1A, whereas in overlap area Ü3 the letter B is almost not visible at all.

[0043] When comparing the similarity of overlapping areas Ü1 and Ü2 to areas Ü1 and Ü3, the image pair 1A and 2A shows greater similarity. For creating a panoramic image, the image pair 1A and 2A would be the preferred combination.

[0044] To determine whether combining image 1A with 1B or image 1A with 2B is more advantageous for the overall panorama, the difference in the overlap area between image 1A and image 1B is measured, as is the difference in the overlap area between image 1A and image 2A. Depending on which of the two differences is smaller, a combination of two images is selected to generate the panorama.

[0045] To determine a difference or difference value in the overlap area of ​​two individual images, one approach is to compare the images pixel by pixel in the overlap area and sum the difference values. Other methods could also be used, such as summing squared difference values ​​determined pixel by pixel, or evaluating a correlation maximum when the pairs of images are shifted relative to each other along the direction in which the visual fields lie adjacent to one another.

[0046] Fig. 5 The figure schematically shows another embodiment in which the individual images are divided into segments or cross-sections, for which a separate decision is made as to whether the sections / cross-sections recorded at the same time are used to generate the panoramic image, or the cross-sections recorded successively, which then do not exhibit parallax.

[0047] The panoramic field of view in this example is characterized by the fact that letter A is subject to movement, i.e., it undergoes a change over time through clockwise rotation, whereas letter B is static. In this case, letter B is positioned above letter A, thus dividing the panoramic image area into an upper area without movement and a lower area with movement.

[0048] In Fig. 5 Therefore, each individual image is divided into an upper and lower area, e.g., individual image 1A is divided into the lower area 1Au and the upper area 1Ao; the same applies analogously to the other individual images 1B, 1C, 2A and 3A.

[0049] When selecting the upper or lower half-images for use in constructing the panorama, the upper area will be discussed first. Due to the lack of movement in the upper area, an analysis essentially follows that of... Fig. 2 This corresponds to the following: Starting from the upper image area 1Ao, the individual image cross-sections 2Ao and 3Ao are preferable to sections 1Bo and 1Co, since the individual image cross-sections 2Ao and 3Ao do not exhibit parallax and are therefore expected to produce a more coherent panoramic image.

[0050] In the lower part of the individual images, it is advantageous in this case to generate the panoramic image based on the individual image cross-sections 1Au, 1Bu, and 1Cu, since the individual image cross-sections 2Au and 3Au exhibit a rotated letter A. Because rotation is often perceived as more disturbing than parallax in human perception, and because the similarities in the overlapping areas are also less pronounced due to the rotation, the parallaxed individual image cross-sections are preferably used for constructing the panoramic image in this case.

[0051] In a case not shown, where the rotation of the letter A occurs significantly more slowly, i.e., where the single image 2Au would show only a slightly rotated letter A, it would be conceivable that in this case the single image cross-section 2Au would have a greater similarity with the corresponding overlap area with the single image cross-section 1Au, and that in this case the combination of 1Au and 2Au would be more advantageous than the parallax-affected combination 1Au and 1Bu; however, it would then also be possible that in the single image 3A the rotation of the letter A is already more clearly perceptible and in this case the combination of 1Au with 1Cu would be preferable to the combination 1Au and 3Au.

[0052] In general, it can be said that the assessment of static scenes, scenes with objects at a relatively large distance, and scenes with objects within a limited distance range is relatively unproblematic.

[0053] Furthermore, it is conceivable to continue the subdivision of individual images into cross-sectional sections, as in Fig. 5 It has been shown that the individual images themselves can be further divided into cross-sections, for example into three, four or even more cross-sectional areas. This can be useful in cases where an individual image is subdivided in a direction perpendicular to the panning direction of the panorama into areas that are either static or exhibit movement.

[0054] To assess whether further subdivision into refined cross-sections is worthwhile, such a subdivision can be performed on a trial basis, and the similarity of the different cross-sections can be calculated. If the respective differences between the cross-sections prove to be insignificant, further refinement into cross-sections can be omitted. If clear differences emerge, it can be concluded that a finer subdivision into cross-sections is worthwhile and can be carried out accordingly. This approach works for a large number of scenes and is only problematic for image sections that contain both motion and depth of field. The depth of field leads to the parallax effects discussed previously.

[0055] The literature describes setups designed to measure and compensate for parallax effects. These setups involve simultaneously capturing individual images with multiple optical channels, each with a fixed viewing direction. Each of these channels is duplicated, with the base width deliberately chosen to be as large as possible. In this way, each partial field of view is captured from two positions. This dual capture of the scene with two cameras at a certain base width / distance allows for the creation of a disparity between these two views. This disparity is then used to compensate for parallax across the entire field of view, ultimately resulting in an artifact-free overall image.

[0056] However, there are scenes where occlusion effects create gaps, either during depth estimation or during compensation. In these cases, there are parts of the image for which depth estimation is either impossible, or for which parallax compensation reveals areas of the scene that were not captured by any of the optical channels. In such cases, parallax compensation is not error-free. Even if the scene is static, an artifact-free or error-free panorama cannot be created. However, if at least one channel is additionally rotated or deflected so that it captures all areas of the field of view, a panoramic image can be created in static scenes without parallax compensation. This is useful in the occlusion case described above, but can also be done without occlusion to avoid the complex parallax compensation process.A device for capturing individual images may include a mirror or other mechanism for deflecting the viewing direction. In this case, the previously described method is particularly advantageous, as the mechanical prerequisites are already in place. Modifications are only necessary in the mirror control or image processing.

[0057] Furthermore, in exemplary embodiments, it may be possible to record all image areas not only with one channel, as in the previously mentioned examples with channel A, but also with the other optical channels B and C. If the viewing directions of the other optical channels are changed simultaneously with the first channel, the recording time is not extended as a result.

[0058] A rapid change in viewing direction, for example to capture individual images 2A and 3A, can be achieved by rotating the entire system or at least the camera system, by using a switchable optical element, such as a tunable lens, a liquid crystal lens, or a liquid lens, or by rotating or tilting a mirror or prism in the beam path. Such a mirror or prism could be driven by a piezoelectric actuator or an electric motor.

[0059] In particular, by using switchable optical elements or by rotating or tilting a mirror or prism, switching times in the range of a few tens of milliseconds can be achieved, which is comparable to typical exposure times in low light conditions.

[0060] In all cases, care must be taken to ensure that the rotation or tilt does not introduce parallax, or at least that it is minimized. It may be sufficient to ensure that the shift caused by parallax is less than the depth of field.

[0061] If the camera system used is equipped with autofocus, all shots can and should be taken with the same focus setting determined during the first shot. This reduces the time between shots to the time required to switch the viewing direction. A consistent focus setting for all individual images is also advantageous for stitching the panorama together.

[0062] Such a camera system according to the invention can be used in mobile phones and vehicle cameras, and can also be used in robotics, videoconferencing systems, remote sensing, medical applications, and machine vision. The method according to the invention is also suitable for multi-channel camera systems in which a mirror or other mechanism for controlling the viewing direction is already present. This is the case, for example, with systems with a folded beam path.

[0063] Although some aspects have been described in connection with a device, it is understood that these aspects also constitute a description of the corresponding process, such that an element of a device can also be understood as a corresponding process step or as a feature of a process step. Similarly, aspects described in connection with or as a process step also constitute a description of a corresponding block, detail, or feature of a corresponding device. Some or all of the process steps can be performed by (or using) a hardware apparatus, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some or more of the key process steps can be performed by such an apparatus.

[0064] Depending on specific implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be carried out using a digital storage medium, for example, a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM, FLASH memory, hard disk, or other magnetic or optical storage medium, on which electronically readable control signals are stored. These control signals can interact with, or interact with, a programmable computer system in such a way as to execute the respective method. Therefore, the digital storage medium can be computer-readable.

[0065] Some embodiments according to the invention therefore include a data carrier which has electronically readable control signals which are able to interact with a programmable computer system in such a way that one of the methods described herein is carried out.

[0066] In general, embodiments of the present invention can be implemented as a computer program product with a program code, wherein the program code is effective in carrying out one of the methods when the computer program product runs on a computer.

[0067] The program code can, for example, also be stored on a machine-readable medium. Other embodiments include the computer program for carrying out one of the methods described herein, wherein the computer program is stored on a machine-readable medium. In other words, an embodiment of the method according to the invention is thus a computer program that includes program code for carrying out one of the methods described herein when the computer program runs on a computer.

[0068] Another embodiment of the methods according to the invention is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for carrying out one of the methods described herein is recorded. The data carrier, the digital storage medium, or the computer-readable medium is typically tangible and / or non-perishable or non-temporary.

[0069] Another embodiment comprises a processing device, for example a computer or a programmable logic device, which is configured or adapted to perform one of the methods described herein.

[0070] Another embodiment comprises a computer on which the computer program for performing one of the procedures described herein is installed.

[0071] In some embodiments, a programmable logic device (for example, a field-programmable gate array, an FPGA) can be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array can interact with a microprocessor to perform one of the methods described herein. Generally, in some embodiments, the methods are performed by any hardware device. This can be general-purpose hardware such as a computer processor (CPU) or method-specific hardware such as an ASIC.

[0072] The devices described herein can be implemented, for example, using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0073] The devices described herein, or any components of the devices described herein, may be implemented at least partially in hardware and / or in software (computer program).

[0074] The methods described herein can be implemented, for example, using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0075] The methods described herein, or any components thereof, may be executed at least partially by hardware and / or by software.

[0076] The embodiments described above merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be obvious to other people skilled in the art. Therefore, it is intended that the invention be limited only by the scope of protection set forth in the following claims and not by the specific details presented herein by way of description and explanation of the embodiments.

Claims

1. Device for generating a panoramic image of a panoramic visual field (150), the device being configured for: simultaneous capturing of a first set of first individual images (1A, 1B, 1C) which together cover the panoramic visual field (150), by means of several optical channels of a first camera system (110); capturing at least one second individual image (2A, 3A) by means of one of the several optical channels of the first camera system (110), so that the at least one second individual image (2A, 3A) results, together or together with a first individual image captured by the one of the several optical channels, in a second set of individual images which are captured with mutual time offset and which cover the panoramic visual field (150); selecting, for at least one image area of the panoramic image, one or more images among a subgroup which is comprised of the first individual images (1A, 1B, 1C) and of the at least one second individual image (2A, 3A) and which overlaps with the at least one image area; and rendering the panoramic image while using the one or more images within the at least one image area; characterized in that the device is further configured for determining, within a first overlap area, a first difference between a first pair of first individual images which relate to at least one portion of the panoramic visual field (150) and which overlap each other, within the panoramic visual field (150), within the first overlap area, determining, within a second overlap area, a second difference between a second pair of individual images from the second set of individual images which relate to the portion of the panoramic visual field (150) and which overlap each other, within the panoramic visual field (150), within the second overlap area, selecting the first pair for rendering the panoramic image within the portion if the first difference is smaller than the second difference, and selecting the second pair for rendering the panoramic image within the portion if the first difference is larger than the second difference.

2. Device as claimed in claim 1, wherein the subgroup includes at least one individual image from the first set and at least the at least one second individual image.

3. Device as claimed in claim 1 or 2, wherein the first individual images are located next to one another along a first direction within the panoramic visual field (150), the device being configured to perform determining of the first and second differences and selecting, on the basis thereof, of the first or second pair along a second direction, which is perpendicular to the first direction, in such a manner that said determining is subdivided into stripes that are transverse to the second direction, so that the selections for different stripes may differ.

4. Device as claimed in any of claims 1 to 3, wherein the first individual images are located next to one another along a first direction within the panoramic visual field (150), the device being configured to select a first group of one or more predetermined images for a first partial image area and to select a second group of one or more predetermined images, which differs from the first subgroup, for a second partial image area which overlaps with the first partial image area along the first direction but is offset from the first partial image area in a second direction perpendicular to the first direction.

5. Device as claimed in any of claims 1 to 4, wherein the first individual images are located next to one another along a first direction within the panoramic visual field (150), the device being configured to perform the selection along a second direction, which is perpendicular to the first direction, in a manner that is subdivided into stripes, so that the selections for different stripes may differ.

6. Device as claimed in claim 3 or 5, configured to adaptively refine the subdivision into stripes.

7. Device as claimed in any of the previous claims, wherein capturing of the at least one second individual image (2A, 3A) is performed such that the second set of individual images is captured with a smaller parallax as compared to the first set of first individual images.

8. Device as claimed in any of claims 1 to 7, wherein capturing of the at least one second individual image (2A, 3A) is performed such that the individual images of the second set of individual images are captured with different viewing directions, and the device being configured to perform, between the individual images of the second set of individual images, a variation in the viewing direction by: rotating the one optical channel or the camera system (110), and / or switching a switchable optical element, which causes deflection of an optical path of the one optical channel or of an optical path of the several channels of the camera system (110); and / or rotating a mirror or prism for deflecting an optical path of the one or more optical channels or an optical path of the several channels of the camera system (110).

9. Device as claimed in any of the previous claims, including a second camera system and further configured for simultaneous capturing of a third set of third individual images (1A, 1B, 1C) which together cover the panoramic visual field (150), by means of several optical channels of the second camera system, the second camera system being arranged offset in relation to the first camera system; capturing at least one fourth individual image (2A, 3A) by means of one of the several optical channels of the second camera system, so that the at least one fourth individual image (2A, 3A) results / result, by itself / by themselves or together with a third individual image captured by the one of the several optical channels, in a fourth set of individual images which are captured with mutual time offset and which cover the panoramic visual field (150); and rendering a further panoramic image or a depth map for the panoramic image on the basis of the third set of third individual images and of the at least one fourth individual image.

10. Device as claimed in claim 9, further configured for estimating a parallax on the basis of the depth map; and compensating for the parallax on the basis of the estimated parallax.

11. Device as claimed in claim 9, further configured for selecting, for the at least one image area of the further panoramic image, one or more images from a subgroup comprised of the third individual images (1A, 1B, 1C) and of the at least one fourth individual image (2A, 3A), said selection for the image area of the further panoramic image corresponding to the selection for the image area of the panoramic image from the subgroup comprised of the first individual images (1A, 1B, 1C) and of the at least one second individual image.

12. Device as claimed in any of claims 1 to 9, including a second camera system and being further configured for simultaneous capturing of a third set of third individual images (1A, 1B, 1C) which together cover the panoramic visual field (150), by means of several optical channels of the second camera system, the second camera system being arranged offset in relation to the first camera system; and compensating for parallax effects within the panoramic image while using the third set of third individual images.

13. Device as claimed in any of claims 1 to 12, further configured for capturing at least one further individual image by means of a further one of the several optical channels of the first camera system (110).

14. Method of generating a panoramic image of a panoramic visual field (150), including: simultaneous capturing of a first set of first individual images (1A, 1B, 1C) which together cover the panoramic visual field (150), by means of several optical channels of a first camera system (110); capturing at least one second individual image (2A, 3A) by means of one of the several optical channels of the first camera system (110), so that the at least one second individual image (2A, 3A) results, together or together with a first individual image captured by the one of the several optical channels, in a second set of individual images which are captured with mutual time offset and which cover the panoramic visual field (150); selecting, for at least one image area of the panoramic image, one or more images among a subgroup which is comprised of the first individual images (1A, 1B, 1C) and of the at least one second individual image (2A, 3A) and which overlaps with the at least one image area; and rendering the panoramic image while using the one or more images within the at least one image area; characterized in that the method further comprises the following steps: determining, within a first overlap area, a first difference between a first pair of first individual images which relate to at least one portion of the panoramic visual field (150) and which overlap each other, within the panoramic visual field (150), within the first overlap area, determining, within a second overlap area, a second difference between a second pair of individual images from the second set of individual images which relate to the portion of the panoramic visual field (150) and which overlap each other, within the panoramic visual field (150), within the second overlap area, selecting the first pair for rendering the panoramic image within the portion if the first difference is smaller than the second difference, and selecting the second pair for rendering the panoramic image within the portion if the first difference is larger than the second difference.