Apparatus and method for capturing images or video

Abstract

An apparatus is provided that includes at least one first sensor and at least one second sensor for capturing first image data, the at least one first sensor and at least one second sensor being rectangular, and the at least one second sensor being arranged orthogonal to the at least one first rectangular sensor; and at least one hardware processor configured to cause the at least one first sensor and at least one second sensor to capture first image data and second image data, respectively, at least substantially simultaneously, and at least one of: display data from the first image data and data from the second image data simultaneously as a cross-shaped image, or store data from the first image data and data from the second image data together as a cross-shaped image. The resulting first image data and second image data can be stored in a single file in memory. The at least one hardware processor can be processed to remove redundancy between the image data. The apparatus can further extract image data from the first image data and the second image data that corresponds to a rectangle parallel to a horizontal line.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to digital photography, and in particular to photography using handheld devices such as smartphones and tablets. BACKGROUND

[0002] This section is intended to introduce the reader to various aspects of art that can be related to various aspects of the present disclosure that are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] Nowadays, when a user takes a picture or a video using a smartphone, he chooses between a portrait format or a landscape format by rotating the smartphone. A picture in portrait format is not suitable for display on a television or other landscape-based display, while, on the contrary, a picture in landscape format is not suitable for use as a wallpaper on a smartphone and for display on a portrait-based display.

[0004] Although the user usually has enough time to choose the appropriate mode (e.g. portrait or landscape), it can also happen that a picture is taken in a hurry due to a sudden event, in which case the user tends to hold the smartphone in the most instinctive way (i.e. in portrait mode) when taking the picture, as these devices are designed to be held in portrait mode with a single hand.

[0005] As far as videos are concerned, most viewing screens are in landscape mode, although people usually record videos in portrait mode using their smartphones, with the result that the videos taken are particularly unsuitable for these viewing screens.

[0006] It can be seen that users taking pictures or recording videos with a smartphone do not always choose the most appropriate mode.

[0007] Therefore, it should be appreciated that it would be desirable to propose a solution to address at least some of the drawbacks related to taking pictures or recording videos with a smartphone. The inventive principle provides such a solution. SUMMARY

[0008] In a first aspect, the present principles relate to a device comprising: at least one first sensor for capturing first image data, the at least one first sensor being rectangular and oriented along a direction with respect to the device; at least one second sensor for capturing second image data, the at least one second sensor being rectangular and oriented along the direction, and being further arranged at least substantially orthogonal to the at least one first rectangular sensor; and at least one hardware processor configured to: cause the at least one first sensor and the at least one second sensor to capture first image data and second image data, respectively, at least substantially simultaneously, and at least one of: simultaneously display data from the first image data and data from the second image data as a cross-shaped image, or store data from the first image data and data from the second image data together as a cross-shaped image.

[0009] In a second aspect, the present principles relate to a method comprising: capturing first image data by at least one first sensor of a device and capturing second image data by at least one second sensor of the device at least substantially simultaneously, the at least one first sensor being rectangular and oriented along a direction with respect to the device, and the at least one second sensor being rectangular and oriented along the direction, and being further arranged at least substantially orthogonal to the at least one first rectangular sensor; and at least one of: simultaneously displaying data from the first image data and data from the second image data as a cross-shaped image, or storing data from the first image data and data from the second image data together as a cross-shaped image.

[0010] In a third aspect, the present principles relate to a computer program product stored on a non-transitory computer readable medium and comprising program code instructions executable by a processor to implement the steps of the method according to any embodiment of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0011] Features of the present principles will now be described, by way of non-limiting examples, with reference to the accompanying drawings, in which:

[0012] Figure 1 A device according to an embodiment of the present principles is shown;

[0013] Figure 2 A method according to an embodiment of the present principles is shown;

[0014] Figure 3A and Figure 3B An example of a capture area according to an embodiment of the present principles is shown;

[0015] Figure 4An example of a file storing a photo is shown, which uses... Figure 2 The method shown captures;

[0016] Figure 5A and Figure 5B Examples of the processed data corresponding to the sensor display are shown respectively;

[0017] Figure 6 A method for processing captured data according to an implementation scheme is shown;

[0018] Figure 7A and Figure 7B Examples of images corresponding to image data and corresponding reduced data are shown;

[0019] Figure 8 An example of a file storing a photo is shown, which uses... Figure 2 The method shown captures and reduces the size;

[0020] Figure 9 A method for capturing video according to an embodiment of the principles of the present invention is shown; and

[0021] Figures 10A to 10D It shows the basis Figure 9 The method uses scroll angle clipping to combine video data. Detailed Implementation

[0022] Figure 1 A device 100 according to an embodiment of the principles of the present invention is shown. Hereinafter, by way of non-limiting example, device 100 will be described as a smartphone; however, it should be understood that device 100 can be implemented as other types of devices, such as tablet computers. Furthermore, where references to images (also referred to as pictures) in the description are made, images can be extended to include videos, which are essentially a set of images.

[0023] The smartphone 100 includes at least one user interface 110, which is configured to receive input from a user, such as instructions and selections, and to provide output to the user. Any suitable user interface can be used, including, for example, a microphone, speaker, haptic actuator, button, keyboard, and touchscreen.

[0024] The smartphone 100 further includes at least one hardware processor 120 (“processor”) configured to control the smartphone 100, process captured images, and execute program code instructions, etc., to perform at least one method according to the principles of the present invention. The smartphone 100 also includes a memory 130 configured to store program code instructions, execution parameters, image data, etc.

[0025] The smartphone 100 further comprises a display 140 configured to output visual information, such as images, possibly captured by the smartphone 100. The display 140 can be a touch screen, which is part of the user interface 110, but is described separately for emphasis.

[0026] The smartphone 100 further comprises at least one first rectangular sensor ("first sensor") 150 and at least one second rectangular sensor ("second sensor") 160 configured to capture images. The first sensor 150 and the second sensor can have the same aspect ratio (e.g. 4 / 3 or 16 / 9), but are oriented to face the same direction, such as directly out of the back of the smartphone, and are respectively orthogonal or at least substantially orthogonal. In other words, one sensor can be oriented to capture portrait images, while the other sensor is oriented to capture landscape images, and possibly have otherwise identical characteristics as the first sensor.

[0027] The smartphone 100 further comprises an angle measurement unit 170 configured to measure a roll angle, i.e. the inclination of the smartphone with respect to the horizontal.

[0028] The smartphone 100 can comprise a plurality of first sensors 150 and a plurality of second sensors 160 oriented to face the same direction, preferably but not necessarily in the same number. As known, among a plurality of sensors, different sensors can have different properties, in order to make it possible to provide, for example, a bokeh effect, a wide-angle function, a telephoto lens function, or an improved fine detail.

[0029] The skilled person will understand that the smartphone will comprise further features, such as a power supply and a radio interface; for the sake of brevity and clarity, only features relevant to the principles of the present invention are discussed.

[0030] When taking a picture, for example in response to a user instruction, the processor 120 will cause both the first sensor 150 and the second sensor 160 to take a picture simultaneously or at least substantially simultaneously. The resulting first and second pictures can be processed and then stored, for example in the memory 130, or transmitted to another device.

[0031] The first and second pictures can be stored in a single file, for example based on the Extensible Device Metadata (XDM) file format, which enables image data from multiple cameras (sensors) to be stored in a single file. However, contrary to the principles of the present invention, conventional use of the XDM format seems to require that the images have the same pose and the same aspect ratio.

[0032] Figure 2 A method 200 of an embodiment of the principles of the present invention is shown. The method 200 can be performed by a device 100 such as the smartphone 100, for example in response to a user instruction. Figure 1the smartphone executes.

[0033] In step S210, the device 100 starts its photo application, thereby enabling the photo function. This can be done for example in response to a user instruction (for example, via the user interface 110 in the smartphone), in response to the execution of software code or in response to an instruction received from an external device. Figure 1

[0034] In step S220, the device 100 displays a capture zone on its display. The capture zone indicates at least substantially the content that can be captured by the first and second sensors of the device. The skilled person will understand that this enables the user to aim the device before taking a picture, in order to frame the picture.

[0035] Figure 3A and Figure 3B Examples of capture zones on the display of a device according to embodiments of principles of the application are shown. Figure 3A A first example of a capture zone with an aspect ratio of 4 / 3 displayed on a smartphone in vertical orientation is shown. Figure 3B A second example of a capture zone with an aspect ratio of 16 / 9 displayed on a smartphone in horizontal orientation is shown.

[0036] As can be seen in Figure 3A and Figure 3B The capture zone is cross-shaped. Indeed, the capture zone shows the content "seen" by the first and second sensors. In an embodiment, the inputs from the first and second sensors are processed (as will be explained below) with the aim to provide a seamless stitching of the inputs. In another embodiment, the capture zone shows the input from one of the sensors and a "missing part" of the input from the other sensor.

[0037] In step S230, the device 100 obtains an instruction to take a picture. As in step S210, the instruction can for example be a user instruction.

[0038] In step S240, in response to the instruction, the device 100 captures the inputs from the first and second sensors, respectively.

[0039] In step S250, the device 100 processes the captured inputs from the first and second sensors. This processing will be further described below.

[0040] In step S260, the device 100 stores the captured photo, i.e. the processed inputs. The captured photo can be stored in the memory of the device or output to another device for storage.

[0041] ​As already mentioned, the captured input can be stored in a single file, for example using the XDM file format, with input from each sensor being associated with a different "camera".

[0042] In embodiments, the processing can comprise associating the captured input from the sensors with sensor pose information (e.g. portrait or landscape) respectively.

[0043] Figure 4 An example of a file storing a photo captured using the method shown in Figure 2 In the example, the file format is XDM.

[0044] As can be seen, the XDM file 400 comprises an indication of the file type 410 (such as JPEG and GIF), XDM device information 420, which comprises device pose 430 and camera information 440. The camera information 440 can comprise information relating to a first camera 442 and information relating to a second camera 444.

[0045] The information relating to the cameras 442, 444 can each correspond to a sensor, for example camera 0 can correspond to the first sensor 150, and camera 1 to the second sensor 160, or vice versa.

[0046] Each of the information relating to the cameras 442, 444 can comprise processed image data 442a, 444a from the corresponding sensor, information about the camera pose (i.e. sensor pose) 442b, 444b and a perspective model 442c, 444c, i.e. the intrinsic parameters of the camera, such as for example focal length, principal point of optical axis, tilt and lens distortion.

[0047] Figure 5A And Figure 5B Examples of the processed data displayed corresponding to the sensors are shown respectively. Figure 5A A landscape image captured by one sensor is shown and Figure 5B A portrait image captured by the other sensor is shown.

[0048] The device, when displaying the stored photo, can show a combined image with data from both sensors combined in any suitable way, either landscape only or portrait only. The user can pre-set or select what the device displays. The device can display the combined image and then, in response to a user instruction, display, store or output one of the two images.

[0049] The skilled person will appreciate that the two images contain a large amount of redundant information and therefore require greater storage requirements for at least some applications.

[0050] In another embodiment, the captured data is processed to reduce the size of the resulting file. It should be noted that such processing can also be used to provide the seamless stitching mentioned with reference to step S220.

[0051] In another embodiment, the image data captured from one sensor is rectified with reference to another sensor, resized, and then cropped to keep only the additional portion of the captured image data, due to the different lenses used for the sensors.

[0052] Figure 6 A method for processing captured data according to another embodiment is illustrated. In this example, while the first camera, camera 0, takes a landscape picture and the second camera, camera 1, takes a portrait picture, this can be the other way around.

[0053] In step S610, the processor obtains the data captured from camera 1, the pose and perspective model from camera 1, and the pose and perspective model from camera 0.

[0054] In step S620, the processor rectifies the image data captured from camera 1 with reference to camera 0, which results in rectified image data. This procedure is well known as it has been described in the following articles: for example, Z. Zhang, "A Flexible New Technique for Camera Calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, November 2000, vol. 22, no. 11, pages 1330-1334, and J.-Y. Bouguet, "Camera Calibration Toolbox for Matlab", available at http: / / www.vision.caltech.edu / bouguetj / calib_doc / index.html, and will therefore not be further described herein.

[0055] In step S630, the processor resizes the rectified image data to match the size of the image data captured from camera 0, which results in resized image data. This procedure is also well known.

[0056] In step S640, the processor removes the redundant portion of the resized image data, which results in reduced image data. The redundant portion can be found in the image data captured from camera 0, which is essentially the overlapping portion.

[0057] In step S650, the processor outputs the reduced image data, for example for storage in a file.

[0058] It should be noted that, should it be desired to use image processing techniques that require data captured from both sensors, such as for example super-resolution imaging, then such image processing should be performed prior to the captured image data being reduced in size using the method in Figure 6

[0059] Figure 7A and Figure 7B An example is shown of an image corresponding to image data and corresponding reduced data. Figure 7A The same landscape image is shown as in Figure 5A while Figure 7B shows additional components (note that there are two additional parts, one corresponding to the part above the image in Figure 7A and one corresponding to the part below).

[0060] Figure 8 An example is shown of a file storing a photograph captured using the method shown in Figure 2 and reduced in size.

[0061] It can be seen that file 800 bears significant similarities to file 400 in Figure 4 ; where the indicated features are the same, the same reference numerals are used. The first difference is that the image data 844a of camera 1 844 includes additional parts, and the second difference is that the image data 844a is associated with reduced image data indicator 844b, which indicates whether the file includes reduced image data, i.e. additional parts, to be combined with the image data of camera 0 prior to display.

[0062] Figure 9 A method for capturing video according to an embodiment of the principles of the application is shown. The method can be performed by the device 100 of Figure 1 ; the reference numerals described in the figures refer to the reference numerals in Figure 1 . It should be noted that the method will typically be implemented as an iterative method, processing one captured image at a time.

[0063] In step S910, the first sensor 150 and the second sensor 160 capture video, i.e. a series of images.

[0064] In step S920, the processor 120 processes the video captured from the second sensor 160 using the method described in Figure 6 , i.e. rectification, resizing and reduction, to generate reduced video data.

[0065] ​In step S930, the processor 120 combines the video from the first sensor 150 with the reduced video data generated in step S920 to generate combined video data.

[0066] In step S940, the processor 120 uses the roll angle, for example measured by the angle measurement unit 170 of the device 100, to crop the combined video data such that the cropped video data is parallel or at least substantially parallel in the landscape mode and in the portrait mode.

[0067] In step S950, the processor 120 outputs the cropped video data to, for example, at least one of the display 140, the memory 130, and an external device.

[0068] Figures 10A to 10D The cropping of the combined video data based on the roll angle in step S940 is illustrated. The figures are intended to show the device 100 rotating from a vertical orientation (also indicated by the vertical hashed rectangle) in a counter-clockwise direction to a near horizontal orientation.

[0069] Each figure shows the combined video (shown as an overlay of the horizontal hashed and vertical hashed rectangles) and the cropped video (shown as a transparent rectangle), and also indicates the roll angle.

[0070] The video is cropped to have the same output size that can be as large as possible while being fully contained in the combined video, regardless of the roll angle. As an example, with each sensor having 1920x1080 pixels, it is possible to obtain a cropped video with pixels 1331x748.

[0071] It will thus be appreciated that the principles of the application can be used to provide a device that is at least to some extent able to compensate for its orientation method when taking pictures or recording videos.

[0072] It is to be understood that the elements illustrated in the figures can be implemented in various forms of hardware, software, or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which can include a processor, a memory, and input / output interfaces.

[0073] The present description illustrates the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its scope.

[0074] All examples and conditional language recited herein are intended to be construed to teach only one example or condition for a process or activity being performed when such examples and conditional language are used in this disclosure and are not intended to necessarily be construed as suggesting that several examples or conditions must exist before the processes or activities performed can be performed.

[0075] Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0076] Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, etc. represent various processes which can be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly illustrated.

[0077] The functions of the various elements shown in the figures can be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions can be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which can be shared. Moreover, explicit use of the term "processor" or "controller" should not be construed to refer exclusively to hardware capable of executing software, and can implicitly include, without limitation, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage.

[0078] Other conventional and / or custom hardware can also be included. Similarly, any switches shown in the figures are conceptual only. Their function can be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selected by the implementer based on his choice and

[0079] In the claims herein, any element expressed as a means for performing a specified function is intended to cover any way of performing that function including, for example, a) a combination of circuit elements that performs that function, or b) software in any form, including, therefore, firmware, microcode, or the like, in combination with the appropriate circuitry for performing that software to perform the function. The disclosures of the present disclosure as defined by such claims encompasses the fact that the functions provided by the various recited means are combined and brought together in the manner required by the claims. Thus, any means that provides those functions are considered to be equivalent to those means shown.

Claims

1. An apparatus for capturing images or videos, the apparatus comprising: A first sensor, which is used to capture first image data corresponding to a first image, is rectangular; A second sensor is used to capture second image data corresponding to a second image, the second image being at least substantially orthogonal to the first image, and the second sensor is rectangular and oriented in the same direction as the first sensor relative to the device. and At least one hardware processor, said at least one hardware processor being configured to: The first sensor and the second sensor capture the first image data and the second image data at least substantially simultaneously, respectively. as well as The data from the first image data and the data from the second image data are stored together as a cross-shaped image.

2. The device of claim 1, further comprising a display configured to display the cross-shaped image.

3. The device of claim 1, wherein the at least one hardware processor is further configured to process at least one of the first image data and the second image data to remove redundancy between the first image data and the second image data.

4. The device of claim 3, wherein the at least one hardware processor is configured to remove redundancy by: The second image data is corrected by referring to the first image data to obtain corrected second image data; The size of the corrected second image data is adjusted to match the size of the first image data, thereby obtaining the second image data with adjusted size; and Remove the portion of the second image data whose size has been adjusted that is present in the first image data.

5. The device according to claim 3, further comprising an angle measuring unit configured to measure the roll angle of the device; The at least one hardware processor is further configured to extract image data from the first image data and the second image data and using the roll angle, the image data corresponding to a rectangular image whose longer edge is parallel to a horizontal line.

6. The device of claim 5, wherein for at least some, but not all, roll angle values, the extracted image data includes image data from both the first image data and the second image data.

7. The device of claim 1, wherein the device further comprises a memory configured to store the data from the first image data and the data from the second image data together.

8. The device of claim 1, wherein the at least one hardware processor is further configured to store the data captured in response to an instruction from the first image data and the data from the second image data in a single file.

9. The device according to claim 1, wherein the first rectangular sensor and the second rectangular sensor have the same aspect ratio.

10. The device of claim 1, wherein the device is a smartphone or tablet computer.

11. The device of claim 1, wherein the second sensor is arranged at least substantially orthogonal to the first sensor.

12. A method for capturing images or videos, the method comprising: At least substantially simultaneously, a first image data corresponding to a first image is captured by a first sensor of the device, and a second image data corresponding to a second image is captured by a second sensor of the device, the second image being at least substantially orthogonal to the first image, the first sensor being rectangular, and the second sensor being rectangular and oriented in the same direction as the first sensor relative to the device; as well as The data from the first image data and the data from the second image data are stored together as a cross-shaped image.

13. The method of claim 12, further comprising processing at least one of the first image data and the second image data by at least one hardware processor to remove redundancy between the first image data and the second image data.

14. The method of claim 13, wherein the at least one hardware processor removes redundancy in the following manner: The second image data is corrected by referring to the first image data to obtain corrected second image data; The size of the corrected second image data is adjusted to match the size of the first image data to obtain a second image data with adjusted size; and Remove the portion of the second image data whose size has been adjusted that is present in the first image data.

15. The method of claim 12, further comprising storing the data captured in response to an instruction from the first image data and the data from the second image data in a single file.

16. The method of claim 12, further comprising: Process at least one of the first image data and the second image data to remove redundancy between the first image data and the second image data; Measure the roll angle of the device; as well as Image data is extracted from the first image data and the second image data using the scroll angle, the image data corresponding to a rectangular image with a longer edge parallel to the horizontal line; For at least some, but not all, roll angle values, the extracted image data includes image data from both the first image data and the second image data.

17. A non-transitory computer-readable medium storing programming code instructions that, when executed by a processor, perform the steps of the method according to any one of claims 12 to 16.

Images