Device and method for capturing images or videos

Abstract

The device includes at least one first sensor and at least one second sensor for capturing first image data, the at least one first and second sensors being rectangular and positioned orthogonal to the at least one first rectangular sensor, and at least one hardware processor configured to cause the at least one first and second sensors to capture first image data and second image data, respectively, at least substantially simultaneously, simultaneously display the data from the first image data and the data from the second image data as a cross-shaped image, and / or store the data from the first image data and the data from the second image data together as a cross-shaped image. The obtained first and second image data can be stored in a single file in memory. The at least one hardware processor can process the image data to remove redundancies. The device can also extract image data corresponding to rectangles parallel to the horizon from the first and second image data.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to digital photography, and more particularly to photography using handheld devices such as smartphones and tablets. [Background technology]

[0002] This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present disclosure, which are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] Currently, when taking photos or videos using a smartphone, the user selects between portrait and landscape format by rotating the smartphone. Portrait format photos are not suitable for display on a television or other landscape-based display, but conversely, landscape format photos are not suitable for use as smartphone wallpaper and for display on a portrait-based display.

[0004] Although users often have enough time to select the appropriate mode (e.g., portrait or landscape), an unexpected event may cause the photo to be taken in a hurry, in which case users tend to take the photo by holding the smartphone in the most intuitive way, i.e., portrait mode, because these devices are designed to be held that way in one hand.

[0005] For video, most viewing screens are in landscape mode, yet people often shoot videos on their smartphones in portrait mode, resulting in videos that are not particularly well suited to these viewing screens.

[0006] Thus, users who take photos or shoot videos with their smartphones do not always select the optimal mode.

[0007] It can therefore be appreciated that a solution is desired that addresses at least some of the drawbacks associated with taking photos or videos with smartphones.The present principles provide such a solution. Summary of the Invention

[0008] In a first aspect, the present principles relate to a device comprising: at least one first sensor for capturing first image data, wherein the at least one first sensor is rectangular and oriented in a direction relative to the device; at least one second sensor for capturing second image data, wherein the at least one second sensor is rectangular and oriented in said direction and further positioned at least approximately orthogonal to the at least one first rectangular sensor; and at least one hardware processor configured to cause the at least one first sensor and the at least one second sensor to at least substantially simultaneously capture first image data and second image data, respectively, and to simultaneously display the data from the first image data and the data from the second image data as a cross-shaped image and / or store the data from the first image data and the data from the second image data together as the cross-shaped image.

[0009] In a second aspect, the present principles relate to a method including at least substantially simultaneously capturing first image data by at least one first sensor of a device and second image data by at least one second sensor of the device, wherein the at least one first sensor is rectangular and oriented in a direction relative to the device, and the at least one second sensor is rectangular and oriented in the direction and further positioned at least approximately orthogonal to the at least one first rectangular sensor; and at least one of simultaneously displaying data from the first image data and data from the second image data as a cross-shaped image or storing data from the first image data and data from the second image data together as a cross-shaped image.

[0010] In a third aspect, the present principles are directed to a computer program product stored on a non-transitory computer-readable medium and comprising program code instructions executable by a processor to implement the steps of the method according to any embodiment of the second aspect. [Brief explanation of the drawings]

[0011] Features of the present principles will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates a device in accordance with an embodiment of the present principles; [Figure 2] 1 illustrates a method in accordance with an embodiment of the present principles. [Figure 3A] 1 illustrates an exemplary capture region in accordance with an embodiment of the present principles; [Figure 3B] 1 illustrates an exemplary capture region in accordance with an embodiment of the present principles; [Figure 4] 3 illustrates an example file storing a photograph captured using the method shown in FIG. 2. [Figure 5A] 10 illustrates an example of displayed processed data corresponding to a sensor. [Figure 5B] 10 illustrates an example of displayed processed data corresponding to a sensor. [Figure 6] 1 illustrates a method for processing captured data according to one embodiment. [Figure 7A] 1 illustrates an example of an image corresponding to image data and the corresponding reduced data. [Figure 7B] 1 illustrates an example of an image corresponding to image data and the corresponding reduced data. [Figure 8] 2 illustrates an example file containing a photograph captured and reduced in size using the method illustrated in FIG. [Figure 9] 1 illustrates a method for capturing video in accordance with an embodiment of the present principles; [Figure 10A] 10 illustrates cropping of the composite video data based on roll angle in the method of FIG. 9. [Figure 10B] 10 illustrates cropping of the composite video data based on roll angle in the method of FIG. 9. [Figure 10C] 10 illustrates cropping of the composite video data based on roll angle in the method of FIG. 9. [Figure 10D] 10 illustrates cropping of the composite video data based on roll angle in the method of FIG. 9. DETAILED DESCRIPTION OF THE INVENTION

[0012] 1 illustrates a device 100 in accordance with an embodiment of the present principles. In the following, by way of non-limiting example, device 100 is described as a smartphone, although it will be understood that device 100 may be implemented as other types of devices, such as a tablet. Additionally, when the description refers to an image (a.k.a., a photograph), this may be extended to include a video, which is actually a set of images.

[0013] Smartphone 100 includes at least one user interface 110 configured to receive input, such as commands and selections, from a user and provide output to the user. Any suitable user interface may be used, including, for example, a microphone, a speaker, a tactile actuator, buttons, a keyboard, and a touchscreen.

[0014] Smartphone 100 further includes at least one hardware processor 120 ("processor") configured to, among other things, control smartphone 100, process captured images, and execute program code instructions to perform at least one method of the present principles. Smartphone 100 also includes memory 130 configured to store program code instructions, execution parameters, image data, etc.

[0015] Smartphone 100 further includes a display 140 configured to output visual information, such as images captured by smartphone 100. Display 140, which may be a touchscreen, is part of user interface 110 but is described separately for emphasis.

[0016] Smartphone 100 further includes at least one first rectangular sensor ("first sensor") 150 and at least one second rectangular sensor ("second sensor") 160 configured to capture images. First sensor 150 and second sensor can have the same aspect ratio (e.g., 4 / 3 or 16 / 9) but face the same direction (e.g., directly outward from the back of the smartphone) and are oriented orthogonal or at least nearly orthogonal to each other. In other words, one sensor can be oriented to capture portrait images, while the other sensor, possibly with the same characteristics as the first sensor, is oriented to capture landscape images.

[0017] The smartphone 100 further includes an angle measurement unit 170 configured to measure the roll angle, i.e. the tilt of the smartphone relative to the horizon.

[0018] Smartphone 100 may include a plurality of first sensors 150 and a plurality of second sensors 160, preferably but not necessarily the same number, oriented to face in the same direction. As is known, in a plurality of sensors, different sensors may have different characteristics to enable, for example, providing bokeh effects, wide-angle capabilities, telephoto lens capabilities, or high definition.

[0019] Those skilled in the art will appreciate that a smartphone may include additional features such as a power source and a wireless interface, etc. For the sake of brevity and clarity, only those features relevant to the present principles will be described.

[0020] For example, when taking a photograph in response to a user command, processor 120 causes both first sensor 150 and second sensor 160 to take a photograph simultaneously, or at least substantially simultaneously. The resulting first and second photographs may be processed, for example, before being stored in memory 130 or transmitted to another device.

[0021] The first and second photographs can be stored in a single file, for example, based on the Extensible Device Metadata (XDM) file format, which allows image data from multiple cameras (sensors) to be stored in a single file. However, contrary to this principle, conventional use of the XDM format appears to require the same pose and the same aspect ratio for the images.

[0022] 2 illustrates a method 200 according to an embodiment of the present principles. Method 200 may be performed by device 100, such as the smartphone of FIG.

[0023] In step S210, device 100 initiates a photo application that enables photo functionality, for example, in response to a user command (e.g., via user interface 110 of FIG. 1), in response to the execution of software code, or in response to commands received from an external device.

[0024] In step S220, device 100 displays a capture area on its display. The capture area at least essentially indicates what may be captured by the device's first and second sensors. Those skilled in the art will appreciate that this allows a user to orient the device and frame the photo before taking it.

[0025] Figures 3A and 3B show examples of capture areas on a device's display, in accordance with an embodiment of the present principles. Figure 3A shows a first example of a capture area with a 4 / 3 aspect ratio displayed on a smartphone in portrait orientation. Figure 3B shows a second example of a capture area with a 16 / 9 aspect ratio displayed on a smartphone in landscape orientation.

[0026] As shown in Figures 3A and 3B, the capture area is cross-shaped. In effect, the capture area represents what is "seen" by the first sensor and by the second sensor. In one embodiment, the inputs from the first and second sensors are processed (as described below) to provide seamless stitching of the inputs. In another embodiment, the capture area represents the input from one sensor and the "missing part" from the input of the other sensor.

[0027] In step S230, device 100 receives an instruction to take a photo. As in step S210, this may be, for example, a user instruction.

[0028] In step S240, device 100 responds to the command to capture inputs from the first and second sensors separately.

[0029] In step S250, device 100 processes the captured inputs from the first and second sensors, as further described below.

[0030] In step S260, the device 100 stores the captured photograph, i.e., the processed input. The captured photograph can be stored in the device's memory or output to a device for further storage.

[0031] As already mentioned, the captured input can be stored in a single file, for example using the XDM file format, with the input from each sensor being associated with a different "camera".

[0032] In one embodiment, processing may include associating each captured input from the sensor with sensor orientation information, such as portrait or landscape orientation.

[0033] Figure 4 shows an example of a file that stores a photograph captured using the method shown in Figure 2. In this example, the file format is XDM.

[0034] As shown, XDM file 400 includes an indication 410 of file types, such as JPEG and GIF, and XDM device information 420, including device pose 430 and camera information 440. Camera information 440 can include information related to a first camera 442 and information related to a second camera 444.

[0035] The camera related information 442, 444 may each correspond to a sensor, for example, camera 0 may correspond to the first sensor 150 and camera 1 may correspond to the second sensor 160, or vice versa.

[0036] Each piece of information 442, 444 related to a camera may include processed image data 442a, 444a from the corresponding sensor, information 442b, 444b related to the camera pose (i.e., sensor pose), and information 442c, 444c related to the perspective model, i.e., camera-specific parameters such as focal length, principal point of the optical axis, skew, and lens distortion.

[0037] Figures 5A and 5B show examples of displayed processed data corresponding to the sensors, respectively: Figure 5A shows a landscape image captured by one sensor, and Figure 5B shows a portrait image captured by the other sensor.

[0038] When displaying a stored photograph, the device can display a composite image in which data from both sensors is combined in any suitable manner, only the landscape image, or only the portrait image. What the device displays can be preset or selected by the user. The device may display the composite image and then, in response to a user command, display, store, or output one of the two images.

[0039] Those skilled in the art will appreciate that the two images contain a lot of redundant information and therefore the storage requirements are greater than necessary for at least some applications.

[0040] In a further embodiment, the captured data is processed to reduce the size of the resulting file. Note that this processing can also be used to provide the seamless stitching described with respect to step S220.

[0041] In a further embodiment, because the lenses for the sensors are not the same, the captured image data from one sensor is corrected and resized relative to the other sensor and then cropped to retain only the additional portion of the captured image data.

[0042] 6 illustrates a method for processing captured data according to a further embodiment. In this example, the first camera, Camera 0, is the camera that took landscape photos and the second camera, Camera 1, is the camera that took portrait photos, although the reverse could also be true.

[0043] In step S610, the processor obtains the captured data from camera 1, the pose and perspective model from camera 1, and the pose and perspective model from camera 0.

[0044] In step S620, the processor corrects the captured image data from camera 1 relative to camera 0, resulting in corrected image data. This is a known procedure, as described, for example, in Z. Zhang, "A Flexible New Technique for Camera Calibration," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, November 2000, and J.-Y. Bouguet, Camera Calibration Toolbox for Matlab, http: / / www.vision.caltech.edu / Boguetj / calib_doc / index.html, and therefore will not be described in detail herein.

[0045] In step S630, the processor resizes the modified image data to match the size of the captured image data from camera 0, resulting in resized image data, which is also a well-known procedure.

[0046] In step S640, the processor removes redundant portions of the resized image data, resulting in a reduced image data. The redundant portions are those found in the captured image data from camera 0, which are essentially overlapping portions.

[0047] In step S650, the processor outputs the reduced image data, for example for storage in a file.

[0048] It should be noted that if it is desirable to use image processing techniques that require captured data from both sensors, such as super-resolution imaging, such image processing should be performed before the captured image data is reduced using the method of FIG.

[0049] Figures 7A and 7B show examples of images corresponding to image data and corresponding scaled-down data. Figure 7A shows the same landscape image as Figure 5A, while Figure 7B shows additional portions (note that there are two additional portions, one corresponding to the upper portion and one corresponding to the lower portion of the image in Figure 7A).

[0050] FIG. 8 shows an example of a file containing a photograph captured and reduced in size using the method shown in FIG.

[0051] As shown, file 800 is substantially similar to file 400 of Figure 4, and the same reference numerals are used where the illustrated features are identical. The first difference is that the image data 844a from camera 1 844 includes additional data, and the second difference is that the image data 844a is accompanied by a reduced image data indicator 844b that indicates whether the file includes reduced image data, which is the additional data that is combined with the image data from camera 0 before display.

[0052] Figure 9 illustrates a method for capturing video in accordance with an embodiment of the present principles. The method may be performed by device 100 of Figure 1. Reference numbers in the figure description refer to reference numbers in Figure 1. Note that the method is typically implemented as an iterative method, processing captured images one at a time.

[0053] In step S910, the first sensor 150 and the second sensor 160 capture a video, ie, a series of images.

[0054] In step S920, the processor 120 processes the captured video from the second sensor 160 using the methods described in FIG. 6, i.e., rectification, resizing, and reduction, to generate reduced video data.

[0055] In step S930, the processor 120 combines the video from the first sensor 150 with the downscaled video data generated in step S920 to generate composite video data.

[0056] In step S940, the processor 120 crops the composite video data, for example using the roll angle measured by the angle measurement unit 170 of the device 100, so that the cropped video data is in a landscape mode parallel to the horizon, or at least approximately parallel to the horizon.

[0057] In step S950, the processor 120 outputs the cropped video data to, for example, at least one of the display 140, the memory 130, and an external device.

[0058] 10A-10D illustrate cropping of the composite video data based on the roll angle in step S940. These figures are intended to illustrate a counterclockwise orientation of device 100 from portrait orientation (also indicated by the vertically striped rectangle) to approximately landscape orientation.

[0059] Each figure shows the composite video (shown as the overlap of a horizontally and vertically striped rectangle) and the cropped video (shown as a transparent rectangle), and also shows the roll angle.

[0060] The video is cropped to have the same output size, which can be as large as possible, while still being fully contained in the composite video regardless of roll angle. As an example, if each sensor has 1920 x 1080 pixels, it may be possible to obtain a cropped video with 1331 x 748 pixels.

[0061] It will therefore be appreciated that the present principles can be used to provide a device that is able to compensate, at least to some extent, for orientation when taking photographs or videos.

[0062] It should be understood that the elements shown in the figures may be implemented in various forms of hardware, software, or a combination thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory, and input / output interfaces.

[0063] The present description illustrates the principles of the present disclosure and it will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the present disclosure and are included within its scope.

[0064] All examples and conditional language recited herein are intended for educational purposes to aid the reader in understanding the principles of the present disclosure and concepts contributed by the inventors to further the art, and should not be construed as being limited to such specifically recited examples and conditions.

[0065] Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. In addition, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0066] Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of the present disclosure. Similarly, it will be appreciated that any flowcharts, flow diagrams, or the like may be substantially represented on a computer-readable medium and represent various processes that may be performed by a computer or processor, whether or not such a computer or processor is explicitly shown.

[0067] The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. If provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. Furthermore, explicit use of the terms "processor" or "controller" should not be construed as referring exclusively to hardware capable of executing software, but may implicitly include, without limitation, digital signal processor (DSP) hardware, read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage.

[0068] Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their functions may be performed through the operation of program logic, through the operation of dedicated logic, through program control and interaction with dedicated logic, or even manually, with the particular technique being selectable by the implementer as more particularly understood from the context.

[0069] In the claims herein, any element expressed as a means for performing a certain function is intended to encompass any way of performing that function, including, for example, a) a combination of circuitry that performs that function, or b) software in any form, including firmware, microcode, etc., combined with appropriate circuitry for executing the software to perform the function. The disclosure as defined by such claims resides in the fact that the functionality provided by the various recited means is combined and brought together in the manner the claims require. Thus, any means that can provide those functionality are regarded as equivalent as shown herein.

Claims

1. A device, at least one first sensor, rectangular in shape, for capturing first image data corresponding to a first image; at least one second sensor for capturing second image data corresponding to a second image that is rectangular and at least substantially orthogonal to the first image, the at least one second sensor being oriented to face in the same direction relative to the device as the at least one first sensor; causing the at least one first sensor and the at least one second sensor to capture the first image data and the second image data, respectively, at least substantially simultaneously; Storing the data from the first image data and the data from the second image data together as a cross image. at least one hardware processor configured to: A device with.

2. The device of claim 1 , further comprising a display configured to display the crosshair image.

3. 2. The device of claim 1, wherein the at least one hardware processor is further configured to process at least one of the first image data and the second image data to remove redundancy between the first image data and the second image data.

4. The at least one hardware processor: modifying the second image data based on the first image data to obtain modified second image data; Resizing the modified second image data to match a size of the first image data to obtain resized second image data; and removing portions of the resized second image data that are present in the first image data; The device of claim 3 configured to remove redundancy by:

5. further comprising an angle measurement unit configured to measure a roll angle of the device; 4. The device of claim 3, wherein the at least one hardware processor is further configured to extract, from the first image data and the second image data, first image data corresponding to a rectangular image whose long sides are horizontal and second image data corresponding to a rectangular image whose long sides are vertical, using the roll angle measured at the time of acquisition of the first image data and the second image data.

6. The device of claim 5 , wherein the extracted image data for at least some but not all roll angle values ​​includes image data from both the first image data and the second image data.

7. A memory configured to store together the data from the first image data and the data from the second image data. The device of claim 1 further comprising:

8. 10. The device of claim 1, wherein the at least one hardware processor is further configured to store the data from the captured first image data and the data from the captured second image data in a single file in response to an instruction.

9. The device of claim 1 , wherein the at least one first sensor and the at least one second sensor have the same aspect ratio.

10. The device of claim 1 , wherein the device is a smartphone or a tablet.

11. The device of claim 1 , wherein the at least one second sensor is positioned at least approximately orthogonal to the at least one first sensor.

12. A step of at least substantially simultaneously capturing first image data corresponding to a first image by at least one first sensor of a device and second image data corresponding to a second image at least approximately orthogonal to the first image by at least one second sensor of the device, wherein the at least one first sensor is rectangular and the at least one second sensor is rectangular and oriented relative to the device to face in the same direction as the at least one first sensor; storing data from the first image data and data from the second image data together as a cross image; A method for providing

13. 13. The method of claim 12, further comprising processing, by at least one hardware processor, at least one of the first image data and the second image data to remove redundancy between the first image data and the second image data.

14. The at least one hardware processor: modifying the second image data based on the first image data to obtain modified second image data; Resizing the modified second image data to match a size of the first image data to obtain resized second image data; and Removing portions of the resized second image data that are present in the first image data.

14. The method of claim 13, wherein redundancy is removed by:

15. 13. The method of claim 12, further comprising storing the data from the first image data and the data from the second image data captured in response to a command in a single file.

16. 16. A non-transitory computer readable medium storing program code instructions which, when executed by a processor, perform the steps of the method of at least one of claims 12 to 15.

Images