Imaging method and imaging device

Abstract

An imaging method and an imaging apparatus (100) are disclosed. A display (3) may be used to display autostereoscopic images to a user. The autostereoscopic image comprises features appearing in an area (1) above the display (3). A detection unit (4) detects a gesture performed by a user in an area (1) above a display. The update unit is configured to update the feature in response to the gesture.

Description

Technical Field

[0001] The present disclosure relates to imaging methods and imaging devices. In particular, the present disclosure relates to displaying an image including features appearing in an area above a display, and detecting gestures performed in the area above the display. Background Art

[0002] Providing touch screens for use by members of the public is very common, with a variety of applications including self-service checkouts in stores. Displays are often used to output information to users and receive input commands from users. The coronavirus pandemic has prompted improvements in the hygiene of infrastructure shared by different people.

[0003] Methods for cleaning infrastructure such as touchscreens have been developed. For many retailers, investing in large quantities of hand sanitizer and disposable paper towels has become standard practice. For these retailers, this has led to increased staff intervention, less efficient checkouts, and a significant increase in biohazard waste. The widely held belief that smartphone automation could provide a solution has been disproven, as research has shown that mobile phone touchscreens themselves are a vector of transmission.

[0004] Contactless human-machine interfaces have been considered more hygienic for users, however, adoption has been limited by cost, integration, and retrofitting issues. In particular, current glasses-free 3D solutions require specialized software applications to generate images and user interfaces. Due to the complexity of 3D calculations, user-facing images must be calculated offline and then recalled in the correct order in response to user input. Currently, there is no contactless device available that can serve as a replacement for touchscreens.

[0005] The present disclosure recognizes that innovation will lie in designing displays that utilize Integral Imaging (InIm), a field of technology originally proposed by G. Lippmann (“ réversibles.Photographics integrates,” CRAcad. Sci. 146, 446–451).

[0006] The integral image is a continuous parallax image that can be recorded to be displayed together with the 2D image using an optical array and observed with the naked eye. The optical array can be a small sheet (microlens array) or a pinhole array or an alternative optical arrangement that performs image reconstruction. The 2D image contains information about the spatial volume, including horizontal and vertical parallax. Current commercial applications of InIm are limited to high-cost professional fields such as microscopy.

[0007] Figure 1BA conventional integral imaging device 10 is shown. The arrangement 10 is suitable for image display or image pickup of a 3D object 11. The arrangement 10 shows a microlens array (MLA) 12 positioned relative to a screen (13, 14). For the image display device, the screen acts as a 2D display 13 having pixels that emit light (e.g., red, green, and blue light for a color display), such that this light from the display 13 is directed by the MLA 12 to provide an image of the 3D object 11. For the image pickup device, the screen acts as a camera 14 having a sensor that is sensitive to light (e.g., red, green, and blue light), such that light is directed from the 3D object 11 to the camera 14 by the MLA 12.

[0008] The reconstructed image is visible in lobes, where the main front lobe typically has a viewing angle of 30-35 degrees. The MLA 12 and the screen (13, 14) are separated by a barrier. Figure 1B The barriers shown are to prevent flipping or sudden image changes and can be expensive to manufacture at the microscopic level.

[0009] Methods for estimating the x, y, and z positions of an object are known. The use of stereo cameras typically involves identifying the object and then using parallax (relative displacement) to estimate depth. Triangulation is used to estimate horizontal and vertical dimensions.

[0010] Identification of foreground objects in an image can be accomplished by comparing them to the background image and can include tracking image differences or motion to establish parallax. The following description primarily refers to objects near the camera being identified as such. However, this arrangement does inherently make identification of such objects easier.

[0011] Existing passive methods using stereo cameras are designed to capture scenes from approximately 1 meter to 100 meters. Outside this range, these methods have poor accuracy. To improve the location of objects near the camera, methods use active sensors such as time-of-flight (ToF) devices. Including a third sensor improves near-field measurements, but cannot accurately handle objects very close to the sensor (a few centimeters) and is also complex and expensive.

[0012] A possible depth estimation technique is to use stereo cameras that are arranged in parallel or slightly angled toward each other. However, this depth estimation is only considered feasible in far-field regions where the parallax decreases with distance. At close distances from the angled cameras, near objects and far objects may be confused. Summary of the Invention

[0013] Various aspects of the invention are set out in the claims.

[0014] This disclosure proposes methods and embodiments that allow the construction of devices that replace conventional touchscreens. Existing applications can be used with a human-machine interface that floats above the screen. 2D images can be suspended at a set distance above the screen, with a touchless pointer function coplanar with the suspended image. All images fed into the device are automatically and seamlessly converted into integral images for display in 3D space.

[0015] As a first aspect, the present disclosure provides an imaging method comprising: displaying an autostereoscopic image to a user, the autostereoscopic image comprising features appearing in an area above a display; detecting a gesture performed by the user in the area above the display; and updating the features in response to the gesture.

[0016] An autostereoscopic image is an integral image configured to be displayed using an optical array.

[0017] Optionally, the feature comprises a virtual button; the gesture comprises selection of the virtual button; and updating comprises providing an indication that the virtual button is selected.

[0018] Optionally, the update includes animation of the autostereoscopic image, wherein the feature appears to move in the area above the autostereoscopic image. This can give the user an indication that the virtual button is being pressed.

[0019] Optionally, the method further comprises generating an autostereoscopic image from the non-autostereoscopic image in real time, thereby enabling video display.

[0020] Optionally, the method further comprises a lookup table to maximize processing efficiency.

[0021] Optionally, the method further comprises modifying the autostereoscopic image in response to the sensor data to present more information in three dimensions and enhance the user experience.

[0022] Optionally, the method further comprises providing a feedback signal consistent with the detected gesture.

[0023] Optionally, the method further comprises generating an autostereoscopic image from the non-autostereoscopic image.

[0024] Optionally, the method further comprises: identifying a foreground of the image; defining an area in the foreground of the image occupied by the moving object; and interpreting whether the moving object is providing a gesture.

[0025] Optionally, the method further comprises detecting a size change and / or a parallax change of the moving object in order to define an area occupied by the moving object.

[0026] As a second aspect, the present disclosure provides a method for detecting a gesture, the method comprising: identifying a foreground of an image; defining an area occupied by a moving object in the foreground of the image; and interpreting whether the moving object is providing a gesture.

[0027] Optionally, identifying the foreground of the image comprises using a plurality of reference lines to identify a bounded volume in front of the detection unit.

[0028] Optionally, the method further comprises performing near-field focusing to distinguish between a first object occupying a foreground of the image and a second object occupying a background of the image.

[0029] Optionally, the method further comprises detecting a size change and / or a parallax change of the moving object in order to define an area occupied by the moving object.

[0030] Optionally, the method further comprises interpreting whether the moving object is providing a gesture, including determining whether the moving object can be used as a pointer.

[0031] Optionally, the method further comprises interpreting whether the moving object is providing a gesture, including performing object localization by reference to data stored by a lookup table or memory.

[0032] The present disclosure provides a combination of the first aspect (which may include any optional features thereof) and the second aspect (which may include any optional features thereof).

[0033] As a third aspect, the present disclosure provides a program that, when executed by a computer, causes the computer to execute the method according to the first aspect or the second aspect.

[0034] As a fourth aspect, the present disclosure provides a computer-readable storage medium storing the program according to the third aspect.

[0035] As a fifth aspect, the present disclosure provides an imaging device comprising: a display configured to display an autostereoscopic image to a user, the autostereoscopic image including features appearing in an area above the display; a detection unit configured to detect a gesture performed by the user in the area above the display; and an update unit configured to update the features in response to the gesture.

[0036] The autostereoscopic image is an integral image; and the display includes an optical array configured to display the integral image.

[0037] Optionally, the optical array has a flat surface configured to face the user.

[0038] Optionally, the optical array covers a portion of the display. The optical array does not cover the entire display. Thus, a first portion of the display is covered by the optical array, and a second portion of the display is not covered by the optical array. The portion of the display covered by the optical array can be used to display a three-dimensional image and / or a two-dimensional image. The portion of the display not covered by the optical array can be used to display a two-dimensional image.

[0039] Optionally, the imaging device further comprises at least two image sensors configured to capture stereoscopic images of objects in the area; wherein the detection unit is configured to detect the gesture by identifying parallax in the stereoscopic images of the objects captured by the at least two image sensors.

[0040] Optionally, the display is configured to display a virtual keyboard.

[0041] Optionally, the imaging device further comprises a card reader configured to read information stored on a card presented by a user.

[0042] Optionally, the imaging device further comprises a feedback unit configured to provide a feedback signal consistent with the gesture detected by the detection unit.

[0043] Optionally, the imaging device further comprises means for generating an autostereoscopic image from the non-autostereoscopic image.

[0044] Optionally, the imaging device further comprises means for generating autostereoscopic images from non-autostereoscopic images in real time, thereby enabling video display and seamless system integration.

[0045] Optionally, the imaging device further includes a lookup table to maximize processing efficiency and reduce power consumption compared to established computational methods.

[0046] Optionally, the imaging device further comprises means for modifying the autostereoscopic image in response to the sensor data, making the device more user friendly, intuitive and improving productivity compared to alternative human machine interfaces.

[0047] As a sixth aspect, a device configured to detect gestures is provided, the device comprising: an identification unit configured to identify the foreground of the image; a delimiting unit configured to delimit the area occupied by a moving object in the foreground of the image; and an interpretation unit configured to interpret whether the moving object is providing the gesture.

[0048] As a seventh aspect, the present disclosure provides an imaging method, which includes generating an autostereoscopic image from a non-autostereoscopic image in real time, thereby realizing video display.

[0049] Optionally, the imaging method further comprises a lookup table.

[0050] Optionally, the imaging method further comprises an autostereoscopic image response to the sensor data.

[0051] As an eighth aspect, the present disclosure provides an imaging device including means for generating an autostereoscopic image from a non-autostereoscopic image in real time to achieve video display.

[0052] Optionally, the imaging device further comprises a lookup table.

[0053] Optionally, the imaging method further comprises an autostereoscopic image response to the sensor data.

[0054] A solution is disclosed that reduces the problem of minimizing the gap g without requiring a barrier. The disclosed software can also reduce this problem. For security applications such as access control systems, the improved viewing angle is unnecessary for privacy reasons. In fact, the system is inherently more secure than a normal keyboard because someone standing next to the user will see different lobes of the user and find it difficult to determine which key was pressed.

[0055] The present disclosure relates to a method for providing the public with displays utilizing integral imaging (InIm). This is achieved using displays comprising mass-produced precision parts coupled with innovative software. These displays allow for the display of images that appear to be in front of the display. This is known as Levitated Photon (trademark) InIm technology and includes the following advantages:

[0056] 1. Generate a realistic optical model in space, resulting in images that are autostereoscopic (without glasses).

[0057] 2. In viewing, eye accommodation and convergence work in unison, meaning that 3D does not rely on an unnatural brain combination of left and right images. At least 15% of the population cannot make this combination, and a significant percentage of those who remain experience headaches.

[0058] 3. Images retain 3D features under monocular observation through accommodation and motion parallax, thus helping the visually impaired.

[0059] 4. The image has horizontal and vertical parallax.

[0060] 5. Compatible with 2D image processing.

[0061] 6. Does not require large amounts of data.

[0062] 7. No potentially dangerous lasers or special projection equipment.

[0063] 8. No special lighting conditions are required for viewing.

[0064] 9. Cost effective and compact.

[0065] 10. Suitable for mass production.

[0066] Existing holographic solutions lack advantages 5, 6, 7, 8, 9, and 10, while existing microlens light field solutions lack advantages 1, 2, 3, 4, 9, and 10.

[0067] The innovative aspects that have been developed include the image creation algorithm, pointer / finger detection method and the overall arrangement that provides low-cost results. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0069] Figure 1A shows a front view of a typical microlens array;

[0070] Figure 1B A conventional integral imaging device is shown;

[0071] Figure 2 An improved integral imaging device is shown;

[0072] Figures 3A to 3D shows views from different angles of a display configured to display stereoscopic images;

[0073] Figure 4 A schematic arrangement of an imaging device is provided;

[0074] Figure 5 shows two parallel image sensors embodied as cameras arranged to create stereo images, along with details of depth estimation and parallax;

[0075] Figure 6 An arrangement for detecting near-field objects co-existing with a 3D image suspended above a display is shown, along with details of the foreground image and foreground disparity;

[0076] Figure 7A A conventional InIm capture or display setup is shown;

[0077] Figure 7B An example showing how to calculate object distances;

[0078] Figure 8A is a flow chart illustrating the implemented imaging method;

[0079] Figure 8B is a flow chart illustrating the implemented pointer detection method;

[0080] Figure 8C is a flow chart illustrating a method for real-time conversion of a non-autostereoscopic image to an autostereoscopic image; and

[0081] Figures 9A to 9C Provide some possible details of the imaging device, where Figure 9A shows the hardware topology; Figure 9B Shows the software topology; and

[0082] Figure 9C Some applications of imaging devices are shown. DETAILED DESCRIPTION

[0083] Various exemplary embodiments, features, and aspects will be described in detail below with reference to the drawings.

[0084] Disclosures are provided for presenting autostereoscopic images to a user.

[0085] Figure 2 An improved integral imaging device 100 is shown, in which the autostereoscopic image is an integral image (InIm). A suspended image 1 is shown, which is created using a microlens array (MLA) 2 to project a 2D image displayed by a display panel 3. The 2D image can appear to be a 3D image to the user. The suspended image 1 is presented to the user, and includes features that appear to occupy an area above the display 3 (i.e., the 3D image appears to be coming out of the screen towards the user). A unique feature is that the 2D image is not binocular, i.e., divided into an area for left eye viewing and an area for right eye viewing.

[0086] The integral imaging device 100 also includes a camera 4 that monitors the area where the suspended image 1 is displayed. The camera 4 is configured to detect movement in the area, such as a gesture performed by a user. The combination of the display integral image (InIm) and finger detection allows user interaction by updating the stereo image based on gestures, so that the 3D features appear to be moved by the user performing the gesture. Existing in-flight control interfaces are not intuitive and are not suitable for security systems where privacy is important.

[0087] Introduced the Float Photon Interface (FPI), which uses the camera's pointer / mouse functionality to produce a touchless screen, such as Figure 2 As shown. Using an acute angle stereo camera, a near field image in front of the screen is captured, and the parallax between the left and right images is used to automatically identify near foreground objects and their distance from the screen. This contactless approach is applicable to both 2D and 3D displays. Multi-touch response is possible, such as an animation of a button moving downward in response to a button press. The microlens array (MLA) 2 has a flat surface facing the user, giving a surface that can be easily dusted and allowing the touch screen to coexist with a suspended photonic interface. This example shows the use of a microlens array 2, but the disclosure extends to the use of any optical component 2 that imparts the functionality of projecting a 2D image being displayed.

[0088] The innovation lies in the way in which in-air button animation is accomplished using foreground / background separation and an image constructed using finger position changes. Various methods are known for constructing and animating stereoscopic images. For example, the left and right images are processed separately using standard techniques before being combined into a viewing format. The animation of the stereoscopic images provides feedback to the user indicating that a gesture has been detected. The imaging device 100 includes a feedback unit that can provide a feedback signal consistent with the detected gesture. Possible feedback units include a speaker configured to provide an audio signal and a haptic device configured to provide a tactile signal. Advantages of these approaches include computational and mechanical simplicity. The wide-angle camera can be located behind a standard flat bezel, without the raised portion around the edge of the screen required for infrared or other beam-interrupting solutions operating in front of the screen.

[0089] The potential of this technology can be demonstrated using a non-optimized arrangement of a 2017 Google phone and a microlens array (MLA). Note that due to the presence of Gorilla Glass (registered trademark) on the OLED display, the 3D image has a limited depth of field. The depth of field is related to the gap g, which is much larger than the optimal value.

[0090] Testing has shown that a floating keypad approximately 25mm above the display is sufficient to prevent users from making contact with the display surface. Furthermore, users have a natural incentive not to overpress, and user feedback is provided when a selection is registered.

[0091] Figures 3A to 3D Shown are views from different angles of a display 3 configured to display an autostereoscopic image 1. The autostereoscopic image shows a levitated cube 1 that appears to be approximately 20 mm above the display 3. Figures 3A to 3D Cube 1 is shown, showing full horizontal and vertical views from four different angles.

[0092] An application of this technology is to install the display 3 in a mobile device such as a smartphone or a card payment reader. The virtual buttons can be arranged to provide a virtual keyboard or virtual keyboard. The display 3 can display one or more virtual buttons that can be selected by the user by performing a gesture of pressing the button with their finger or stylus. The user typically indicates a button selection by performing a press gesture by moving their finger toward the display and then moving their finger away from the display.

[0093] The user's selection of the virtual button is confirmed by the display presenting an animation showing the button being pressed. The animation indicates to the user that a selection gesture is being performed. As the finger moves toward the display, the button appears to move from a first position farther from the display to a second position closer to the display. As the finger moves away from the display, the button appears to move from the second position closer to the display back to the first position farther from the display. Thus, the image is updated to indicate that the virtual button has been selected.

[0094] Security applications such as card payment readers often include a numeric keypad. An 11m keyboard will typically be arranged with keys 4, 5, 6 arranged horizontally and keys 9, 6, 3 and a decimal point arranged vertically.

[0095] Figure 4 A schematic arrangement of an imaging device 100 is shown. A suspended image 1 is shown, created using optical components 2 to image a flat screen display 3. The imaging device 100 includes application software 5, a display image processor 6, and a front image processor 7. Optionally, an input / output (I / O) unit 8 is used to provide connectivity to any number of live feeds 9 and / or any number of memory storage devices 10. Note that the components shown can be implemented as hardware, firmware, software, or a mixture of all three.

[0096] The hovering image 1 provides a 3D image of the 2D integral image generated by the display 3. The hovering 3D image 1 appears at varying depths to illustrate an animation in which a push gesture is depicted by the user's finger moving toward the display. Many methods for creating integral images are well known, including using third-party modeling tools such as Autodesk 3ds Max (registered trademark). However, Figure 8C The method described in represents the fastest and most efficient way to create an integral image from a non-integral image. Thus, a levitated 3D image 1 is generated by hardware, firmware and software components, wherein the user presses a levitated button that animates a downward push with a finger.

[0097] The optical component 2 is arranged in front of the display 3. The optical component 2 is typically a lens array, such as a micro lens array (MLA), or a pinhole array, which is configured to construct the 3D image 1 from the integral image (InIm) displayed by the display 3.

[0098] The flat screen display 3 may or may not include a touch screen for displaying the integral image (InIms) generated by the application software 5, the display image processor 6, the front image processor 7, or any combination of the three. Optionally, the touch screen may be integrated into the optical component (e.g., micro lens array (MLA)) 2.

[0099] Two or more cameras 4 are positioned to capture a near-field view of the volume surrounding the suspended image 1. The captured content is fed from the cameras 4 to a front image processor 7 for processing. The content may include stereoscopic or depth information. To optimize scene capture, the cameras 4 can be electronically or mechanically rotated or repositioned to an optimal angle. Optionally, the cameras have a focal length and depth of field that can also be changed optically or electronically.

[0100] The application software 5 determines the overall function that the device 100 performs for the user, such as the device 100 being used as a credit card reader. The application software 5 determines what functions are to be performed by the display image processor 6, the front image processor 7, and the I / O unit 8. Thus, the application software 5 controls the configuration of the display 3 to present an integral image to the user based on input received from the front image processor 7 or via the I / O unit 8.

[0101] The display image processor 6 is configured to process the image of the display 3 based on instructions received from the application software 5. The source image information can come directly from the front image processor 7, or from the live feed 9 and / or storage device 10 via the I / O connection unit 8. One process is to convert a real-time (video rate) 2D image into InIms, which is perceived as a 2D image suspended at a set distance from the display 3. Another is a real-time conversion of a 2D image plus depth (2DD) to InIms, which is perceived as a 3D suspended image. An additional process can change the InIms generated in real time in response to input from the front image processor 7. This allows a portion of the constructed image to change in depth in response to a press event, thereby enabling button animation.

[0102] The front image processor 7 is configured to process images received from the camera 4 to enable a contactless human-machine interface by identifying objects close to the optical component 2 and calculating their 3D position. This near-field technology is superior to existing visual tracking and artificial intelligence (AI) camera solutions due to its simplicity. The front image processor 7 can also process 2D and 3D scene images and video for conversion to InIms by the display image processor 6 or direct transmission to the live feed 9 and / or memory storage device 10.

[0103] The connection is achieved using an input / output (I / O) unit 8 which may implement an internal software process or USB, wireless or other established interface technology.

[0104] The live stream input from one or more live feeds 9 is real-time video (2D, 2DD or 3D) or images streamed to the display image processor 6 for processing. The live stream output to one or more live feeds 9 is real-time video (2D, 2DD or 3D) or images that have been processed by the display image processor 6 and streamed out.

[0105] The storage device 10 for storage may be solid-state or non-solid-state, and may be used to store programs, processed images and videos, and source images and videos.

[0106] While traditional ray tracing techniques and third-party software can convert 3D images into integral images, these techniques are slow for real-time operation and require offline processing of the image to be ready for display when needed. This is understandable, considering that the entire volume of 3D space may have to be processed for each microlens in the microlens array. The following details a method for real-time conversion of 2D, 2DD, or 3DD images into integral images.

[0107] Figure 7A A conventional InIm capture or display setup is shown. In capture mode, an optical component (72) comprising a lens array in front of an image sensor (71) is used to capture a volume (74) consisting of voxels (75). The component can be hardware or software or a combination of both. The value of each voxel at a particular location in the volume is represented by S(x, y, z). The value of each pixel in the image sensor or display is represented by T(x I , Y I ) represents. S(x, y, z) and T(X I , Y I ) are typically red, green, and blue (RGB) values.

[0108] In the proposed embodiment, light optics are used to track each voxel position to a position on (71) through each lens (73). Each lens (73) has a corresponding pixel area (76) on the image sensor (71). All voxel positions are defined in InIm because InIm is not resolution-limited and is independent of the display resolution. Position equivalence can be expressed as S(x, y, z) ≡ T(X I , Y I )≡S(X I , Y I ). Given the volume and microlens specifications, a lookup table can be generated offline. This table can be used for real-time operation because ray tracing calculations, such as Figure 8C shown.

[0109] For display purposes, the resolution of InIm is limited by the display panel and needs to be downsampled. This can be, but is not limited to, rounding off or truncating or interpolating to the nearest voxel from surrounding voxels.

[0110] General principles

[0111] An image presented by a display includes a foreground and a background. The foreground area is near the display. A camera is configured to detect a moving object. If the foreground is occupied by a moving object, a region of the image including the moving object is defined. This region is defined based on detecting changes in size and / or parallax of the moving object. The image detected by the camera is interpreted to assess whether the moving object is providing a gesture.

[0112] Figure 5 Two parallel cameras are arranged to create stereo images, along with details of depth estimation and parallax. The parallel cameras have a wide viewing angle of 2xθ0. Objects 51-54 are located in front of the cameras. Object 54 is in the foreground, while the other objects are in the background.

[0113] Conventional depth estimation shows images from the left and right cameras. For the purpose of this position description, objects are shown as having the same size and not decreasing with distance from the camera. Furthermore, it is assumed that the objects have already been identified using existing processing equipment.

[0114] For conventional methods, it can be seen that object 54 occludes object 53 in image L and object 51 in image R. If objects 53 and 51 are both similar to object 54, the viewer will not know whether object 54 or objects 51 and 53 are being seen. Images L and R are combined along parallel camera centerlines to give a disparity image, from which disparity can be calculated using standard processing. It can be seen that the disparity increases for objects moving from the position of object 52 in the background to the position of object 54 in the foreground.

[0115] The new method rotates the parallax reference line inward by θ1. Image L and image R remain the same, however, the combined parallax image has new characteristics. This time, the parallax of an object moving from the position of object 52 to the position of object 54 decreases to zero (0 parallax), and then increases as it moves closer to the camera.

[0116] For this new approach, the redefined reference lines intersect at zero, and this defines the boundary between the background and foreground. It can also be said that any object that appears in both the region (θ0-θ1) of image L and the region (θ0-θ1) of image R must be in the foreground unless it coincides with two similar objects in the background at the same angular position. Although the object sizes are shown as equal in the figure, the change in object size with motion is another indication of whether the object is in the foreground or background. In the background, objects moving towards the camera increase in size but decrease in parallax until they reach the background boundary. Conversely, objects in the foreground moving towards the camera will increase in size and increase in parallax. Table 1 below shows the situation.

[0117] Table 1:

[0118] size Parallax Location <![CDATA[Relative to the camera motion M p > Increase Increase prospect near Reduce Increase background keep away Increase Reduce background near Reduce Reduce prospect keep away constant constant constant still

[0119] Pointer function

[0120] Figure 6 An arrangement for detecting near-field objects co-existing with a 3D image suspended above a display is shown. Figure 6 Details of the foreground image and foreground disparity are also shown.

[0121] In this case, the camera is set at an angle that makes the parallax reference line coincide with the camera centerline to maximize the detection volume and reduce the maximum parallax for close objects, thereby minimizing supersensitivity at close distances.

[0122] The camera has a viewing angle of approximately 80 degrees, which is typical for small, low-cost cameras, and the depth of field is set to focus only on the foreground area. The 3D image is not registered by the camera because the light from the display constructing the 3D image is assumed to be projected away from the camera.

[0123] Object 55 has been added to the 3D display volume. Objects 54 and 55 are both in the foreground qualification area of images L and R. For clarity, objects 51 and 53 are not shown in images L and R. In any case, since the background is outside the camera's depth of field and is therefore out of focus and has low contrast, the likelihood that objects 51 and 53 could give a foreground misreading is greatly reduced.

[0124] The foreground disparity shows the disparity of the object 54 and the object 55 that enable depth detection in the foreground space when the objects move from outside to inside the 3D volume.

[0125] Current depth estimation methods using disparity are often unreliable and inconsistent. Table 1 allows the relative motion of objects to be used to verify the object position determined by the disparity calculation. This, together with the near-field camera setup, increases the overall reliability of the method and reduces misinterpretations.

[0126] Pointer-triggered measurement

[0127] The position of an object, such as object 55, can be determined by triangulation. The x, y, and z object distances can be calculated using the known parameters of camera separation and field of view angle.

[0128] Figure 7B An example showing how to calculate the x, y, and z object distances. The camera has a 90-degree field of view. A and B are the centers of the camera lenses, and C is the zero point. A is the reference 0, 0, 0 position. The spacing between the cameras is I ABOften referred to as interocular distance, but in this case the camera spacing is not intended to replicate human eye separation, as is typically used in current stereo cameras. The generalized example shown is for a camera that is vertically in an elevated position and located at X p 、Y p and Z p Object P at a location similar to the location of object 55.

[0129] Image L and Image R show the camera images at the camera resolution. The relative position of the object P in these images will be used to determine the real-world distance X p 、Y p and Z p .U VL and V HL are the horizontal and vertical resolutions of the left camera, U HR and V VR are the horizontal and vertical resolutions of the right camera. The position of object P in image L and image R is represented by U PL 、U PR and Y p Its position can also be expressed as a ratio of its maximum size. In the case of cameras with similar specifications, the position of the object P is given by U PL / U H 、U PR / U H and V PL / V V This method makes the calculation independent of the camera resolution, as long as there is a minimum resolution that can meet the desired granularity of the design.

[0130] To calculate X p 、Y p and Z p The equation for is shown below. In its simplest form, there are only three variables involved in the calculation, utilizing standard trigonometric functions.

[0131] Disparity can be expressed as a ratio of horizontal and vertical camera resolutions, making the calculation of the position of the object of interest independent of the image sensor resolution and simpler. This is suitable for real-time implementation with a lookup table. In this way, it is possible to detect a finger above the display and then initiate and process real-time 3D image changes to implement functions such as button press animation with finger movement.

[0132] Pointer calibration

[0133] Pointer calibration can be used as an alternative or in addition to using triangulation to determine the x, y, z position of an object. When used to supplement triangulation, reference objects are placed throughout the volume of interest, and their actual positions are compared with the calculated positions to generate error correction values applicable to objects found in or around the space occupied by the reference objects.

[0134] If sufficient reference objects are used together with known positions to record U PL 、U PR and V PL or the values of its derivatives, a high-resolution map can be created to serve as a lookup table for accurate pointer operations.

[0135] Calculation of the 3D position of object P

[0136] For horizontal resolution, U HL =U HR =U H (similar specs camera).

[0137] For symmetrically positioned cameras, θ B =θ A =θ H and is half the horizontal viewing angle. AB It is also known.

[0138] Using triangulation

[0139] X p =I AB x tan(θ H -θ PR ) / (tan(θ H -θ PL )+tan(θ H -θ PR )) Equation 1

[0140] in

[0141] θ PL =tan -1 (tanθ H x 2U PL / U H ) Equation 2

[0142] and

[0143] θ PR =tan -1 (tanθ H x 2U PR / U H ) Equation 3

[0144] Therefore, the horizontal position X is calculated using the ratio of the horizontal width p , making the calculation independent of the camera resolution.

[0145] Z p =tan(θ H -θ PL )x X p Equation 4

[0146] For vertical resolution, V VL =V VR =V v (Cameras with similar specifications)

[0147] For similar cameras, θ v It is half of the vertical viewing angle of the two cameras.

[0148] Using triangulation

[0149] R PL =Z p / Sin(θ H -θ PL ) Equation 5

[0150] Yp=(tanθ v x2V PL / V v )(Z p / Sin(θ H -θ PL )) Equation 6

[0151] Therefore, the vertical position Y is calculated using the ratio of the vertical height p , making the calculation independent of the camera resolution.

[0152] A minimum of three variables U can be used PL 、U PR and V PL To determine the 3D position of the object P entering the near-field space.

[0153] Implementation of imaging methods

[0154] Figure 8A is a flow chart illustrating the implemented imaging method S100 .

[0155] In step S101, an image is acquired for display on the display 3. The image can be acquired in a variety of ways, including from a stored memory, communication, image processing, or from a computer program according to a predetermined set of parameters. Figure 4 A combination of one or more sources arranged as shown. Step S102 determines whether the image is suitable for display based on the requirements of the application software 5. If the image is displayed in step S104, the image is first displayed using the image in step S103. Figure 9A and Figure 9B The topology of the image or a specified portion is converted into an autostereoscopic image. Thus, a stereoscopic image can be displayed to the user. The present disclosure provides an autostereoscopic image generated from a non-autostereoscopic image. The present disclosure also provides for displaying an autostereoscopic image that includes features that appear in an area above the display 3.

[0156] In step S105, the camera 4 is set up with parameters such as frame rate, focal length, aperture, field of view, resolution and orientation angle. This allows the pointer operation to be optimized for the required operating conditions.

[0157] In steps S106 to S109, any gesture performed by the user in the area above the display may be detected. Figure 8B The detection method S200 is described in detail in the flowchart of FIG.

[0158] In step S106, the images captured by the camera 4 are monitored and any objects of interest are detected. This can be done by comparing consecutive video frames, establishing differences of interest, and using motion relative to the camera, as shown in Table 1.

[0159] In step S107, the horizontal and vertical center positions are determined for any object detected in the image captured by the camera 4. One method of calculating the center position is to take the average of the maximum and minimum points occupied by the object for both axes. This method works well for objects with uniform shapes, such as fingers.

[0160] In step S108, it is evaluated whether any object has been detected in the foreground of the image that has been captured. If one or more objects have been detected in the foreground of the image, the method proceeds to step S109. On the other hand, if no object has been detected in the foreground of the image that has been captured, the method returns to step S106.

[0161] In step S109, the position of the object is determined. This can be done by calculating the position of the object. Alternatively, this can be done by finding the position of the object.

[0162] In step S110, an evaluation is made as to whether any objects are present in the 3D image volume. If one or more objects are present in the 3D image volume, the method proceeds to step S111. On the other hand, if no objects are present in the 3D image volume, the method returns to step S106.

[0163] In step S111, an evaluation is made as to whether a system response is to be performed. Examples of system responses are connection to a website, operation of a mechanical device such as a door, powering off a device, or user feedback such as an audible beep. If a system response is to be performed, the method proceeds to step S112. On the other hand, if no response is to be performed, the method returns to step S106.

[0164] In step S112, the system executes the response or specified instruction.

[0165] In step S113, it is evaluated whether a display response is to be performed. If a display response is to be performed, the method proceeds to step S114. On the other hand, if a display response is not to be performed, the method returns to step S106.

[0166] In step S114, a display response is performed. Thus, the 2D and / or 3D image displayed by the display 3 is changed or animated by returning to step S101. For example, a button may change color and / or move downward with the user's finger. Thus, the image is updated in response to the gesture.

[0167] Figure 8C is a flowchart showing the real-time conversion method of S103.

[0168] Step S103.1 determines the format of the input image, which can be a standard 2D image, a 2D image with depth information, a point cloud 3D image, or other 3D formats.

[0169] Once the format is known, the volume parameters are determined in step S103.4 based on the user settings from S103.3. The user can specify additional parameters such as depth, as is often required for purely 2D source images. The user can also specify modification parameters such as magnification factors and position offsets. Another example is depth inversion for changing an artifact image into a normal image.

[0170] Modification parameters may also be derived from sensors in S103.2, such as from an accelerometer, gyroscope, or magnetometer. For example, in response to a non-optimal orientation of the screen, volume parameters may be modified to tilt and / or rotate the 3D image toward the user. Another example is using thermometer data to change the 3D size of a feature in response to temperature changes.

[0171] Step S103.5 transforms the source image based on the volume parameters from S103.4 to produce an object image volume in voxels, which will be described as an integral image in subsequent steps.

[0172] In step S103.6, the 3D volume from S103.5 is mapped onto the 2D display surface using a lookup table to determine equivalent positions.

[0173] Step S103.7 sends the voxel value, typically designated as red, green, and blue data, for use in its equivalent integral Figure 2 D position, provided that the position has not been previously used by another voxel in the 3D volume. For example, duplication may occur if one voxel is behind another voxel through line of sight.

[0174] In such cases, step S103.8 is used for voxel / pixel processing. Step S103.8 decides how to resolve mapping conflicts based on the user settings in S103.3, resulting in whether the result is used or not. The user can prioritize voxels closest to the viewer, in which case the depth of the voxel will determine whether the voxel is mapped. Another use case example is where the user wants to smooth out rounding errors and use voxel values interpolated from similarly mapped or surrounding voxels.

[0175] Step S103.9 combines the integral maps for display on a voxel-by-voxel or pixel-by-pixel basis. The mapping is repeated until the target integral map is completed.

[0176] For video, S103 is repeated at the frame rate (e.g., 60 fps). Note that this process can be completed without any calculation.

[0177] Figure 8C The real-time approach shown is equally applicable to a range of 3D display solutions, such as pinhole and lenticular display systems.

[0178] Use the pointer method to detect gestures. The characteristics of this pointer method are:

[0179] 1. Use converging guides to form a bounded volume in front of the camera to mathematically define the foreground / background boundary.

[0180] 2. Use the changes in object size and parallax to define the area occupied by the moving object and use this information to verify the object.

[0181] 3. Detect near-field objects that can be used as pointers.

[0182] 4. Camera near-field focus and depth of field settings avoid confusion with background objects.

[0183] 5. A method of object positioning (x, y, z) that is independent of camera and display resolution, which is easily implemented by storing data in memory or lookup tables.

[0184] Figure 8B is a flow chart showing the implemented pointer detection method S200 .

[0185] S200 illustrates a method of calculating the pointer position at runtime, however, the equations mentioned may be used to generate a lookup table that may be stored in memory to achieve the same result.

[0186] Once the camera is set up in S105, step S201 enters the camera parameters. These parameters form the basis for the pointer position.

[0187] Step S202 tracks the size and motion of the object to determine if it is an object of interest according to Table 1. In near-field detection, this is an object that meets the foreground condition. If an object of interest exists, the method proceeds to S203, otherwise repeating step S202.

[0188] Before proceeding to step S204, step S203 obtains the parallax parameters and relative motion M of the object of interest P. p .

[0189] Step S204 uses Figure 6 The position of the object P in the foreground area is confirmed by the disparity of . If this is not confirmed, the method returns to S202, otherwise S200 proceeds to S205 and S206, where the left horizontal object angle and the right horizontal object angle are calculated using Equation 2 and Equation 3, respectively.

[0190] In S207, S208 and S209, the horizontal angle θ PL and θ PR Used to calculate the object P coordinate X using equations 1, 6, and 4 p 、Y p , Z p .

[0191] Step S210 uses the difference between the current position and the previous position of the object P to determine the relative motion of P and compares it with the M obtained from S203. p If there is consistency, the method proceeds to S212, otherwise an error is detected and handled in step S211. Before returning to the monitoring step S202, the valid coordinates are output for use by the system in S212.

[0192] Uses of the present invention

[0193] For many arrangements, the display 3 is used to present both the 2D image and the 3D integral image.A 3D arrangement may present to the user a plurality of virtual buttons arranged to form a virtual keyboard or virtual keyboard.

[0194] This can be achieved by applying MLA 2 to the lower half of the screen, which for the card reader application device 100 allows 3D images to be presented in the secure customer area, such as a floating keyboard. Therefore, the disclosure provides an MLA that covers a portion of the display area, which allows 3D images to be presented in that portion. Note that MLA 2 can cover the entire screen and still present a 2D image because the height and depth of each voxel can be set in software. The present disclosure covers credit card readers that comply with EMVCo Certified (registered trademark).

[0195] The innovation lies in the use of the disclosed device 100 for keypad entry, health, and safety systems. In this case, as well as the contactless advantage, the keypad can be located behind a pane of glass or window, where the keypad can be operated from another room or in front of glass outside. This is inherently more secure because the entry system can be implemented without external wiring or mechanical attachments.

[0196] The present disclosure provides software that has been developed to model any 3D full color image, icon or video, real world or graphic for 3D visualization in space without glasses.

[0197] Figure 9A The hardware topology of the imaging device 100 is shown to include a microlens array 2, an OLED (organic light emitting diode) display 3, a first camera (camera 1) and a second camera (camera 2), a sensor, an input / output (I / O) unit 8, one or more processors, one or more memories, and a voltage regulation unit. Possible I / O units may include any combination of an HDMI port, a USB1 port, a USB 2 port, a USB 3 port, and a Wi-Fi receiver (registered trademark). The device 100 is shown to include a central processing unit (CPU), a field programmable gate array, and firmware. Note that it is not necessary for the hardware topology to include these features, for example, the HDMI interface can be eliminated by using Intel Thunderbolt technology (registered trademark, Intel Thunderbolt port). Processing can be performed externally, such as offline calculation of source images, which provides an opportunity to manufacture lower-cost non-real-time products.

[0198] Figure 9B It is shown that the software topology of the imaging device 100 may include application software, libraries and network-based interfaces, hardware abstraction layers, drivers, and a kernel.

[0199] Figure 9CSeveral applications of the imaging device 100 are shown. Applications can be used with displays 3 of varying sizes. Applications implementing small-size displays include access control systems, elevator controls, and appliances. Applications implementing medium-size displays include fuel station equipment, parking meters, ATMs (automated teller machines), and vending machines. Applications implementing large-size displays include airport check-in equipment, information kiosks, ordering systems, point-of-sale devices, self-service checkout devices, and healthcare devices.

[0200] The disclosed algorithms, software and optics developments offer advantages over expensive holographic and light field solutions and utilize standard components.

[0201] Thus, the present disclosure provides a human-machine interface presented in space such that there is no user contact and therefore no surface pathogen transmission. The Float Photon interface eliminates detergents, waste, and human intervention while increasing throughput and being carbon negative.

[0202] The disclosed examples can be implemented by a computer or a system or a device. These examples can be implemented by any device configured to execute instructions or any dedicated hardware capable of performing all or part of the functions. The present disclosure provides hardware (for example, a processor such as a central processing unit (CPU) or a microprocessor unit (MPU)) that is configured to read out and execute a program recorded on a memory device to perform the functions of the disclosed examples. To this end, a program is provided to a computer, for example, via a network or from various types of recording media used as a memory device (for example, a computer-readable medium such as a non-transient computer-readable medium). The steps of the disclosed method can be performed in any suitable order, or simultaneously where possible.

[0203] Although the disclosed subject matter has been described using specific terms relating to apparatus features and / or method features, it should be understood that the claimed subject matter is not necessarily limited to the disclosed examples. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions. The advantages disclosed may relate to several of the disclosed examples.

[0204] Clauses

[0205] The following clauses form part of the description and correspond to the claims as filed in priority document GB2218713.2.

[0206] 1. An imaging method, comprising:

[0207] displaying an autostereoscopic image to a user, the autostereoscopic image including features appearing in an area above the display;

[0208] detecting a gesture performed by the user in the area above the display; and updating the feature in response to the gesture.

[0209] 2. The imaging method of clause 1, wherein:

[0210] The autostereoscopic image is an integral image configured to be displayed using a microlens array.

[0211] 3. The imaging method of clause 1 or clause 2, wherein:

[0212] The features include virtual buttons;

[0213] The gesture comprises a selection of the virtual button; and

[0214] The updating includes providing an indication that the virtual button is selected.

[0215] 4. The imaging method according to any one of the preceding clauses, wherein:

[0216] The updating comprises animation of the autostereoscopic image wherein the feature appears to move in the area above the display.

[0217] 5. The imaging method according to any of the preceding clauses, further comprising:

[0218] Provides feedback consistent with the detected gesture.

[0219] 6. The imaging method according to any preceding clause, further comprising: generating an autostereoscopic image from a non-autostereoscopic image.

[0220] 7. The imaging method according to any of the preceding clauses, further comprising:

[0221] identifying a foreground of the image;

[0222] defining an area occupied by a moving object in the foreground of the image; and

[0223] Interpreting whether the mobile object is providing the gesture.

[0224] 8. The imaging method of clause 7, further comprising:

[0225] A size change and / or a parallax change of the moving object is detected in order to define the area occupied by the moving object.

[0226] 9. A method for detecting a gesture, the method comprising:

[0227] identifying a foreground of the image;

[0228] defining an area occupied by a moving object in the foreground of the image; and

[0229] Interpreting whether the mobile object is providing the gesture.

[0230] 10. The method of clause 9, wherein:

[0231] Identifying a foreground of the image includes using a plurality of reference lines to identify a bounded volume in front of a detection unit.

[0232] 11. The method of clause 9 or clause 10, further comprising:

[0233] Near-field focusing is performed to distinguish a first object occupying the foreground of the image from a second object occupying a background of the image.

[0234] 12. A method according to any one of clauses 9 to 11, further comprising:

[0235] A size change and / or a parallax change of the moving object is detected in order to define the area occupied by the moving object.

[0236] 13. A method according to any one of clauses 9 to 12, wherein:

[0237] Interpreting whether the moving object is providing the gesture includes determining whether the moving object can function as a pointer.

[0238] 14. A method according to any one of clauses 9 to 13, wherein:

[0239] Interpreting whether the mobile object is providing the gesture includes performing object localization by referencing data stored by a lookup table or memory.

[0240] 15. A program which, when executed by a computer, causes the computer to perform the method according to any preceding clause.

[0241] 16. A computer-readable storage medium storing the program according to clause 15.

[0242] 17. An imaging device comprising:

[0243] a display configured to display an autostereoscopic image to a user, the autostereoscopic image including features appearing in an area above the display;

[0244] a detection unit configured to detect a gesture performed by the user in the area above the display; and

[0245] An updating unit is configured to update the feature in response to the gesture.

[0246] 18. The imaging apparatus of clause 17, wherein:

[0247] The autostereoscopic image is an integral image; and

[0248] The display includes a microlens array configured to display the integral image.

[0249] 19. The imaging apparatus of clause 19, wherein:

[0250] The microlens array has a flat surface configured to face a user.

[0251] 20. The imaging apparatus of clause 18 or clause 19, wherein:

[0252] The microlens array covers a portion of the display.

[0253] 21. The imaging device of any one of clauses 17 to 20, further comprising: at least two cameras configured to capture stereo images of objects in the area;

[0254] Wherein the detection unit is configured to detect the gesture by identifying disparity in the stereoscopic images of the object that have been captured by the at least two cameras.

[0255] 22. An imaging apparatus according to any one of clauses 17 to 21, wherein:

[0256] The display is configured to display a virtual keyboard.

[0257] 23. The imaging device of any of clauses 17 to 22, further comprising a card reader configured to read information stored on a card presented by the user.

[0258] 24. The imaging device according to any one of clauses 17 to 23, further comprising: a feedback unit configured to provide a feedback signal consistent with the gesture detected by the detection unit.

[0259] 25. Imaging apparatus according to any of clauses 17 to 24, further comprising means for generating an autostereoscopic image from a non-autostereoscopic image.

Claims

1. An imaging method comprising: displaying an autostereoscopic image to a user, the autostereoscopic image including features appearing in an area above the display, wherein the autostereoscopic image is an integral image configured for display using the optical array; detecting a gesture performed by the user in the area above the display; as well as The characteristic is updated in response to the gesture.

2. The imaging method according to claim 1, wherein: The features include virtual buttons; The gesture comprises a selection of the virtual button; and The updating includes providing an indication that the virtual button is selected.

3. The imaging method according to claim 1 or claim 2, wherein: The updating comprises animation of the autostereoscopic image wherein the feature appears to move in the area above the display.

4. The imaging method according to any one of the preceding claims, further comprising: Generate autostereoscopic images from non-autostereoscopic images in real time for video display.

5. The imaging method according to any one of the preceding claims, further comprising: Lookup table.

6. The imaging method according to any one of the preceding claims, further comprising: The autostereoscopic image is modified in response to sensor data.

7. The imaging method according to any one of the preceding claims, further comprising: Provides feedback consistent with the detected gesture.

8. The imaging method according to any one of the preceding claims, further comprising: Generate autostereoscopic images from non-autostereoscopic images.

9. The imaging method according to any one of the preceding claims, further comprising: identifying a foreground of the image; defining an area occupied by a moving object in a foreground of the image; as well as Interpreting whether the mobile object is providing the gesture.

10. The imaging method according to claim 9, further comprising: A size change and / or a parallax change of the moving object is detected in order to define the area occupied by the moving object.

11. The imaging method according to claim 9 or claim 10, wherein: Identifying the foreground of the image includes using a plurality of reference lines to identify a bounded volume in front of the detection unit.

12. The imaging method according to any one of claims 9 to 11, further comprising: Near-field focusing is performed to distinguish between a first object occupying the foreground of the image and a second object occupying a background of the image.

13. A program which, when executed by a computer, causes the computer to carry out the method according to any one of the preceding claims.

14. A computer-readable storage medium storing the program according to claim 13.

15. An imaging device comprising: a display configured to display an autostereoscopic image to a user, the autostereoscopic image comprising features appearing in a region above the display, wherein the autostereoscopic image is an integral image and the display comprises an optical array configured to display the integral image; a detection unit configured to detect a gesture performed by the user in the area above the display; as well as An updating unit is configured to update the feature in response to the gesture.

16. The imaging device according to claim 15, wherein: The optical array has a flat surface configured to face a user.

17. The imaging device according to claim 15 or claim 16, wherein: The optical array covers a portion of the display.

18. The imaging device according to any one of claims 15 to 17, further comprising: At least two image sensors are configured to capture stereoscopic images of objects in the area. The detection unit is configured to detect the gesture by identifying parallax in the stereoscopic images of the object captured by the at least two image sensors.

19. The imaging device according to any one of claims 15 to 18, wherein: The display is configured to display a virtual keyboard.

20. The imaging device according to any one of claims 15 to 19, further comprising: A card reader is configured to read information stored on a card delivered by the user.

21. The imaging device according to any one of claims 15 to 20, further comprising: A feedback unit is configured to provide a feedback signal consistent with the gesture detected by the detection unit.

22. The imaging device according to any one of claims 15 to 21, further comprising: Apparatus for generating an autostereoscopic image from a non-autostereoscopic image.

23. The imaging device according to any one of claims 15 to 22, further comprising: An apparatus for generating autostereoscopic images from non-autostereoscopic images in real time for video display.

24. The imaging device according to any one of claims 15 to 23, further comprising: Lookup table.

25. The imaging device according to any one of claims 15 to 24, further comprising: Means for modifying the autostereoscopic image in response to sensor data.

26. An imaging method comprising: Generate autostereoscopic images from non-autostereoscopic images in real time for video display.

27. The imaging method according to claim 26, further comprising: Lookup table.

28. The imaging method of claim 26 or claim 27, further comprising: Autostereoscopic image response to sensor data.

29. An imaging device comprising: An apparatus for generating autostereoscopic images from non-autostereoscopic images in real time for video display.

30. The imaging device according to claim 29, further comprising: Lookup table.

31. The imaging device according to claim 29 or claim 30, further comprising: Autostereoscopic image response to sensor data.

Images