DEVICE AND METHOD FOR CAPTURING IMAGES OR VIDEO

Abstract

A device includes at least one first sensor and at least one second sensor for capturing first image data. The first and second sensors are rectangular and arranged orthogonally to each other. The device also includes at least one hardware processor configured to process the first and second sensors' data at least substantially simultaneously, and to simultaneously display the first and second image data as a cross-shaped image or to jointly store the first and second image data as a cross-shaped image. The resulting first and second image data can be stored in a single file in memory. The hardware processor can be configured to eliminate redundancies between the image data.In addition to the first and second image data, the device can also extract image data corresponding to a parallel rectangle with a horizon.

Description

DEVICE AND METHOD TO CAPTURE IMAGES OR VIDEO FIELD OF INVENTION This description generally relates to digital photography and in particular to photography with handheld devices such as smartphones and tablets. BACKGROUND OF THE INVENTION This section is intended to introduce the reader to various aspects of the technique, which may relate to several aspects of the present description described and / or claimed later. It is believed that this discussion is helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present description. Accordingly, these statements should be understood to be read in this context and not as admissions of prior art. Currently, when taking photos or videos with a smartphone, the user selects between portrait and landscape formats by rotating the phone. A portrait photo is not suitable for viewing on a television or other screen in landscape format, while, conversely, a landscape photo is unsuitable for use as a smartphone wallpaper or for viewing on portrait-format screens. Ref. 337169 While the user often has enough time to select an appropriate mode (e.g., portrait or landscape), it may happen that a photograph is taken on the fly due to an unexpected event, in which case the user tends to take the picture holding the smartphone in the most instinctive way, i.e., in portrait mode, because these devices are designed to be held that way in one hand. In the case of video, most display screens are in landscape mode, but despite that, people often record video in portrait mode with their smartphone, which makes the video not particularly suitable for these display screens. As can be seen, users who take pictures or record videos with smartphones do not always choose the most appropriate mode. Therefore, it will be appreciated that there is a desire for a solution that addresses at least some of the shortcomings related to taking photographs or recording video with a smartphone. These principles provide such a solution. BRIEF DESCRIPTION OF THE INVENTION In one respect, the present principles are directed to a device comprising at least a first sensor for capturing first image data, the at least one QfrRRnn / zznz / E / YiAi first sensor is rectangular and directed in a direction relative to the device, at least one second sensor for capturing second image data, the at least one second sensor is rectangular and directed in the direction and arranged further at least essentially orthogonal to the at least one first rectangular sensor, and at least one hardware processor configured to make the at least one first sensor and the at least one second sensor capture, respectively, the first image data and the second image data at least substantially simultaneously, and at least one of simultaneously displaying data from the first image data and data from the second image data as a cross-shaped image or jointly storing data from the first image data and data from the second image data as a cross-shaped image. In a second aspect, the present principles are directed to a method comprising capturing at least substantially simultaneously the first image data by at least one first sensor of a device and the second image data by at least one second sensor of the device, the at least one first sensor being rectangular and directed in a direction relative to the device, and the at least one second sensor being rectangular and directed in the direction and arranged further at least essentially orthogonal to the at least one first rectangular sensor, at least one of simultaneously displaying data from the first image data and data from the second image data as a cross-shaped image or jointly storing data from the first image data and data from the second image data as a cross-shaped image. In a third aspect, the present principles are directed to a computer program product that is stored on a non-transient, computer-readable medium and includes program code instructions executable by a processor to implement the steps of a method in accordance with any modality of the second aspect. BRIEF DESCRIPTION OF THE FIGURES The characteristics of the present principles will now be described, by way of non-limiting example, with reference to the attached figures, in which: Figure 1 illustrates a device in accordance with one modality of the present principles; Figure 2 illustrates a method in accordance with one modality of the present principles; Figures 3A and 3B illustrate examples of capture zones according to modalities of the present principles, Figure 4 illustrates an example of a file that stores a photograph captured using the method illustrated in Figure 2; QfrRRnn / zznz / E / YiAi Figures 5A and 5B illustrate examples of the visualized processed data corresponding to the sensors; Figure 6 illustrates a method for processing captured data according to a modality; Figures 7A and 7B illustrate examples of images that correspond to image data and corresponding reduced data; Figure 8 illustrates an example of a file that stores a photograph captured using the method illustrated in Figure 2 and that has been reduced in size; Figure 9 illustrates a method for capturing video according to one modality of the present principles; and Figures 10A-10D illustrate the cropping of the combined video data based on the balance angle in the method of Figure 9. DETAILED DESCRIPTION OF THE INVENTION Figure 1 illustrates a device 100 according to one embodiment of these principles. Hereafter, as a non-limiting example, device 100 will be described as a smartphone, but it is understood that device 100 can be implemented as other types of devices, such as a tablet. Furthermore, where the description mentions images (also referred to as photographs), this can be extended to include video, which in practice is a set of images. QfrRRnn / zznz / E / YiAi The smartphone 100 includes at least one user interface 110 configured to receive input, such as instructions and selections, from a user and provide output to the user. Any suitable user interface may be used, including, for example, a microphone, a speaker, a haptic actuator, buttons, a keyboard, and a touchscreen. The smartphone 100 further includes at least one hardware processor 120 (processor) configured to, among other things, control the smartphone 100, process captured images, and execute program code instructions to perform at least one method of the present principles. The smartphone 100 also includes memory 130 configured to store program code instructions, execution parameters, image data, etc. The smartphone 100 also includes a display 140 configured to produce visual information such as images, possibly captured by the smartphone 100. The display 140, which may be a touch screen, is part of the user interface 110, but is described separately for emphasis. The smartphone 100 further includes at least one first rectangular sensor (first sensor) 150 and at least one second rectangular sensor (second sensor) 160 configured for image capture. The first sensor 150 and the second sensor may have the same aspect ratios (e.g., 4:3 or 16:9), but they are oriented in the same direction (such as directly outward from the back of the smartphone) and are respectively orthogonal or at least essentially orthogonal. In other words, one sensor may be oriented for portrait photography while the other sensor, possibly in any other way with the same characteristics as the first, is oriented for landscape photography. The smartphone 100 also includes an angle measuring unit 170 configured to measure the balance angle, i.e., the tilt of the smartphone relative to the horizon. The smartphone 100 may include a plurality of first sensors 150 and a plurality of second sensors 160, preferably, but not necessarily, the same number, oriented in the same direction. As is known, in a plurality of sensors, different sensors may have different properties to make it possible to offer, for example, bokeh effects, wide-angle capability, telephoto lens capability, or enhanced fine detail. An expert in technology will understand that a smartphone will include additional features such as QfrRRnn / zznz / E / YiAi a power supply and radio interfaces; for reasons of brevity and clarity, only the characteristics related to the present principles are discussed. When an image is taken, for example, in response to user instructions, the processor 120 will cause both the first sensor 150 and the second sensor 160 to take an image simultaneously, or at least substantially simultaneously. The resulting first image and the second image can then be processed before being stored, for example, in memory 130, or transmitted to another device. The first and second images can be stored in a single file, for example, based on the Extensible Device Metadata (XDM) file format, which allows image data from multiple cameras (sensors) to be stored in a single file. Contrary to current principles, however, conventional use of the XDM format appears to require the same orientation and aspect ratio for the images. Figure 2 illustrates one method 200 of a modality of the present principles. Method 200 can be performed by a device 100 such as the smartphone in Figure 1. At stage S210, device 100 starts its QfrRRnn / zznz / E / YiAi is a photographic application that enables photographic functionality. This can be done, for example, in response to user instructions (e.g., via user interface 110 in Figure 1), in response to the execution of software code, or in response to instructions received from an external device. In the S220 stage, the device displays a capture zone on its screen. This capture zone indicates, at least essentially, what can be captured by the device's first and second sensors. Experts will appreciate that this allows users to aim the device before taking a photo, to frame the image. Figures 3A and 3B illustrate examples of capture zones on a device screen according to the modalities of these principles. Figure 3A illustrates a first example of a capture zone with a 4:3 aspect ratio displayed on a vertically oriented smartphone. Figure 3B illustrates a second example of a capture zone with a 16:9 aspect ratio displayed on a horizontally oriented smartphone. As can be seen in Figures 3A and 3B, the capture area is cross-shaped. In fact, the capture area displays what is seen by the first sensor and by the second sensor. In one mode, the input from the first sensor and the second sensor are processed (as will be explained later). QfrRRnn / zznz / E / YiAi ahead) with the aim of providing a homogeneous union of the inputs. In another mode, the capture area shows the input of one of the sensors and the missing pieces of the input of the other sensor. In stage S230, device 100 receives instructions to take an image. As in stage S210, this could be, for example, user instructions. In stage S240, in response to the instructions, device 100 captures the input from the first and second sensors separately. In stage S250, device 100 processes the input captured from the first and second sensors. This processing will be described in detail later. In stage S260, the device 100 stores the captured photograph, i.e., the processed input. The captured photograph can be stored in the device's memory or sent to an external storage device. As already mentioned, the captured input can be stored in a single file, e.g., using the XDM file format, from each sensor input that is associated with a different camera. In one mode, processing may include respectively associating the captured sensor input with sensor posture information, for example, vertical or horizontal. Figure 4 illustrates an example of a file that stores a photograph captured using the method illustrated in Figure 2. In the example, the file format is XDM. As can be seen, the XDM 400 file includes a file type indication 410, such as JPEG and GIF, XDM device information 420 which includes the device position 430, and camera information 440. The camera information 440 may include information related to a first camera 442 and information related to a second camera 444. The information related to a camera 442, 444 may each correspond to a sensor, e.g., camera 0 may correspond to the first sensor 150 and camera 1 to the second sensor 160, or vice versa. Each piece of information related to a camera 442, 444 may include processed image data 442-a, 444a from the corresponding sensor, information about the camera's posture (i.e., the sensor's posture) 442b, 444b, and a perspective model 442c, 444c, i.e., intrinsic parameters of a camera such as, for example, focal length, principal point for the optical axis, skew, and lens distortion. Figures 5A and 5B illustrate examples of the processed data displayed, corresponding to the sensors. QfrRRnn / zznz / E / YiAi Figure 5A shows a landscape mode image captured by one sensor and Figure 5B shows a portrait mode image captured by the other sensor. When viewing a stored photograph, the device can display a combined image using data from both sensors in any suitable way, only the landscape image, or only the portrait image. A user can pre-configure or select what the device displays. The device can display the combined image and then, in response to user instructions, display, store, or generate one of the two images. The expert will appreciate that the two images contain highly redundant information and that the storage requirements are therefore greater than those needed for at least some applications. In an additional mode, the captured data is processed to reduce the size of the resulting file. It is noted that this processing can also be used to provide the homogeneous join mentioned with reference to stage S220. In the additional mode, because the lenses for the sensors are not the same, the image data captured from one sensor is rectified to the reference of the other sensor, resized and then cropped to keep only the additional parts of the captured image data. Figure 6 illustrates a method for processing captured data according to the additional mode. In this example, the first camera, camera 0, took the photo in landscape mode and the second camera, camera 1, took the image in portrait mode, but it can be the other way around. In stage S610, a processor obtains the captured data from camera 1, the posture and perspective model of camera 1, and the posture and perspective model of camera 0. In stage S620, the processor rectifies the image data captured from camera 1 to the reference of camera 0, resulting in rectified image data. This is a well-known procedure that will not be described in detail here, as it has been described in, for example, Z. Zhang, "A Flexible New Technique for Camera Calibration," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330–1334, Nov. 2000, and in J.-Y. Bouguet, "Camera Calibration Toolbox for Matlab," http: / / www.vision.caltech.edu / bouguetj / calib_doc / index.html. In stage S630, the processor resizes the rectified image data to match the size of the image data captured from camera 0, resulting in resized image data. This is also a well-known procedure. In stage 3640, the processor removes redundant portions from the resized image data, resulting in reduced image data. These redundant portions are those found in the image data captured from camera 0, essentially overlapping areas. In stage 3650, the processor generates the reduced image data, for example, for file storage. It is observed that, in case it is desired to use image processing techniques that require the captured data from both sensors, such as, for example, super-resolution imaging, then image processing must be performed before the captured image data is reduced using the method in Figure 6. Figures 7A and 7B illustrate example images that correspond to image data and corresponding reduced data. Figure 7A illustrates the same landscape-mode image as in Figure 5A, while Figure 7B illustrates the additional parts (note that there are two additional parts, one corresponding to the top of the image in 7A, and one corresponding to the bottom). Figure 8 illustrates an example of a file that stores a photograph captured using the illustrated method. QfrRRnn / zznz / E / YiAi in Figure 2 and which has been reduced in size. As can be seen, file 800 bears a significant resemblance to file 400 in Figure 4; the same reference numbers are used when the stated characteristics are identical. A first difference is that the image data 844a from camera 1 includes the additional parts. A second difference is that the image data 844a is associated with a reduced image data indicator 844b, which indicates whether or not the file includes reduced image data—that is, additional parts—to be combined with the image data from camera 0 before display. Figure 9 illustrates a method for capturing video according to one modality of the present principles. The device 100 in Figure 1 can perform the method; the reference numbers in the figure description refer to those in Figure 1. Note that the method would typically be implemented as an iterative method that processes one captured image at a time. In the S910 stage, the first 150 sensor and the second 160 sensor capture video, that is, a series of images. In the S920 stage, the processor 120 processes the video captured from the second sensor 160 using the method described in Figure 6, i.e., rectification, resizing and reduction, to generate reduced video data. QfrRRnn / zznz / E / YiAi In the S930 stage, the processor 120 combines the video from the first sensor 150 with the reduced video data generated in the S920 stage to generate combined video data. In the S940 stage, the processor 120 uses a balance angle, e.g., measured by the angular measuring unit 170 of the device 100, to crop the combined video data in such a way that the cropped video data is in landscape mode parallel or at least essentially parallel to the horizontal. In the S950 stage, the processor 120 generates the cropped video data, for example, to at least one of the screen 140, memory 130 and an external device. Figures 10A-10D illustrate the cropping of the combined video data based on the balance angle at the S940 stage. The figures are intended to illustrate the left-hand orientation of the device 100 from vertical (also indicated by the vertically marked rectangle) to near horizontal. Each figure illustrates the combined video (shown as an overlay of a horizontally marked rectangle and a vertically marked rectangle) and the cropped video (shown as a transparent rectangle), and also indicates the balance angle. The video is cropped to maintain the same output size, which can be as large as possible while remaining fully contained within the combined video, regardless of the balance angle. For example, if each sensor has a resolution of 1920x1080 pixels, it may be possible to obtain a cropped video of 1331x748 pixels. Therefore, it will be appreciated that the present principles can be used to provide a device capable of, at least to some degree, compensating for orientation when taking a photograph or recording a video. It should be understood that the elements shown in the figures can be implemented in various forms of hardware, software, or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor interface, memory, and input / output. The present description illustrates the principles of this document. Therefore, it will be appreciated that those skilled in the art may devise various arrangements that, while not explicitly described or shown herein, incorporate the principles of this document and fall within its scope. All examples and conditional language mentioned in this description are intended for educational purposes to help the reader understand the principles of the description and the concepts contributed by the inventor to the development of the technique; they should be interpreted without being limited to such examples and conditions specifically mentioned. Furthermore, all statements in this description that mention principles, aspects, and modalities of the description, as well as specific examples thereof, are intended to encompass both their structural and functional equivalents. Moreover, such equivalents are intended to include both currently known equivalents and equivalents developed in the future; that is, any developed element that performs the same function, regardless of structure. Therefore, for example, those skilled in the art will appreciate that the block diagrams presented herein represent conceptual views of illustrative circuits that incorporate the principles of the description. Similarly, it will be appreciated that any flowcharts, flow diagrams, and the like represent various processes that can be substantially represented on computer-readable media and thus executed by a computer or processor, regardless of whether such a computer or processor is explicitly shown or not. The functions of the various elements shown The functions QfrRRnn / zznz / E / YiAi in the figures can be provided by the use of dedicated hardware, as well as hardware capable of running software in conjunction with appropriate software. When provided by a processor, the functions can be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared. Furthermore, the explicit use of the term processor or controller should not be construed as referring exclusively to hardware capable of running software, and may implicitly include, but is not limited to, digital signal processor (DSP) hardware, read-only memory (ROM) for storing the software, random-access memory (RAM), and non-volatile storage. Other hardware, both conventional and / or custom, may also be included. Similarly, any switches shown in the figures are purely conceptual. Their function may be performed through program logic, dedicated logic, the interaction of program control and dedicated logic, or even manually; the specific technique is selectable by the implementer, as best understood from the context. In the claims of this description, QfrRRnn / zznz / E / YiAi Any element expressed as a means of performing a specified function is intended to encompass any way of performing that function, including, for example, a) a combination of circuit elements performing that function, or b) software in any form, including firmware, microcode, or the like, combined with a circuit system suitable for executing that software to perform the function. The description as defined by these claims lies in the fact that the functionalities provided by the various means mentioned are combined and joined in the manner described in the claims. Therefore, any means capable of providing these functions is deemed equivalent to those shown in the present description. It is hereby stated that, as of this date, the best method known to the applicant for putting the aforementioned invention into practice is the one that is clear from the present description of the invention.

Claims

Having described the invention as above, the following claims are claimed as property: 1.A device characterized in that it comprises: at least one first sensor for capturing first image data corresponding to a first image, the at least one first sensor being rectangular; at least one second sensor for capturing second image data corresponding to a second image at least substantially orthogonal to the first image, the at least one second sensor being rectangular and oriented in the same direction as the at least one first sensor relative to the device; and at least one hardware processor configured to: cause the at least one first sensor and the at least one second sensor to capture, respectively, the first image data and the second image data at least substantially simultaneously; and store together the first image data and the second image data as a cross-shaped image.

2. The device according to claim 1, characterized in that it further comprises a screen configured to display the image in the shape of a cross.

3. The device according to claim 1, characterized in that the at least one hardware processor is further configured to process at least one of the first image data and the second image data to eliminate redundancies between the first image data and the second image data.

4. The device according to claim 3, characterized in that the at least one hardware processor is configured to eliminate redundancies by: rectifying the second image data with reference to the first image data to obtain rectified second image data; resizing the rectified second image data to match a size of the first image data to obtain resized second image data; and removing portions of the resized second image data present in the first image data.

5. The device according to claim 3, characterized in that it further comprises an angle measuring unit configured to measure a balance angle of the device; wherein the at least one hardware processor is further configured to extract, from the first image data and the second image data and using the balance angle, image data corresponding to a rectangular image whose longer edge is parallel to the horizon.

6. The device according to claim 5, characterized in that the extracted image data, for at least some, but not all, balance angle values, includes image data from both the first image data and the second image data.

7. The device according to claim 1, characterized in that it further comprises memory configured to store together the data of the first image data and the data of the second image data.

8. The device according to claim 1, characterized in that the at least one hardware processor is further configured to store the first image data and the second image data, captured in response to an instruction, in a single file.

9. The device according to claim 1, characterized in that the at least one first rectangular sensor and the at least one second rectangular sensor have the same aspect ratio.

10. The device according to claim 1, characterized in that it is a smartphone or a tablet.

11. The device according to claim 1, characterized in that the at least one second sensor is arranged at least essentially orthogonal to at least one first sensor.

12. A method characterized in that it comprises: capturing at least substantially simultaneously first image data corresponding to a first image by means of at least one first sensor of a device and second image data corresponding to a second image at least essentially orthogonal to the first image by means of at least one second sensor of the device, the at least one first sensor being rectangular, and at least one second sensor being rectangular and oriented in the same direction as the at least one first sensor relative to the device; and storing together data from the first image data and data from the second image data as a cross-shaped image.

13. The method according to claim 12, characterized in that it further comprises processing, by means of at least one hardware processor, at least one of the first image data and the second image data to eliminate redundancies between the first image data and the second image data. QfrRRnn / zznz / E / YiAi 14. The method according to claim 13, characterized in that the at least one hardware processor eliminates redundancies by: rectifying the second image data with reference to the first image data to obtain rectified second image data; resizing the rectified second image data to match a size of the first image data to obtain resized second image data; and removing portions of the resized second image data present in the first image data.

15. The method according to claim 12, characterized in that it further comprises storing the first image data and the second image data, captured in response to an instruction, in a single file.

16. A non-transient, computer-readable medium, characterized in that it stores program code instructions which, when executed by a processor, implement the steps of a method in accordance with QfrRRnn / zznz / E / YiAi less one of claims 12 to 15.

Images