Non-contact gesture commands for touch screens

Abstract

A method for displaying aerial images includes: detecting a capacitive response to a non-contact gesture made by a user on a proximity touch-sensing enabled display screen, and, in response to detecting the non-contact gesture, displaying one or more images in a three-dimensional (3D) space in proximity to the display screen.

Description

[0001] This application is a divisional application of Chinese patent application filed on February 9, 2021, with application number 202180013579.3 and invention title "Non-contact gesture commands for touch screen". Technical Field

[0002] Embodiments of this disclosure generally relate to display devices equipped with touchscreens, and more particularly to the detection of non-contact gestures made above such devices for controlling the aerial display of images in 3D space above such devices. Background Technology

[0003] In traditional AR, display devices equipped with both touchscreens and display functionality (such as, for example, in-vehicle displays, smartphones, tablets, laptops, etc.) project both real-world images and user interface (UI) graphics onto a 2D display screen. The real-world images can be captured, for example, by a camera, and the UI graphics are provided by the AR application. In this traditional AR mode, to view the combined (augmented) image, the user must look directly at the 2D display to see the AR object displayed alongside the real-world image. If the input device is provided in a car, to see the road and also interact with the display, the user must frequently move their gaze between the 2D display of the input device and a 3D view of the real world seen through the vehicle's windshield and mirrors.

[0004] Furthermore, there are other contexts where users of display devices (equipped with touchscreen functionality) desire to use AR applications but cannot physically touch the display screen. For example, there might be cooking apps that teach the technology by adding AR icons, graphics, and messages to real-world images of ingredients—for instance, adding a "cut here" arrow to an image of an onion, or showing the preferred consistency of a sauce right next to an actual pan. Users of such apps expect to interact with them while cooking but often have hands too dirty to touch the screen. Additionally, such users don't want to have to look at the display, as it's inconvenient to cook while looking at a screen.

[0005] Display devices are needed to address these issues of traditional AR. Summary of the Invention

[0006] The present invention is provided in a simplified form to describe the selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0007] In one embodiment, a method for displaying aerial images is disclosed. The method includes detecting a capacitive response to a non-contact gesture made by a user on a proximity-touch-sensing-enabled display, and in response to detecting the non-contact gesture, displaying one or more images in a three-dimensional (3D) space in proximity to the display.

[0008] In another embodiment, a system for aerial display of images is also disclosed. The system includes a display screen and a touchscreen sensing system, the display screen including a 3D imaging apparatus, and the touchscreen sensing system configured to acquire sets of capacitive responses to sets of corresponding non-contact gestures made by a user approaching the display screen. The system also includes a processor coupled to each of the display screen and the touchscreen sensing system, the processor being configured to control the 3D imaging apparatus to display one or more images in an aerial display mode in 3D space approaching the display screen.

[0009] In another embodiment, a method for interactively displaying an image in a 3D space near a display device is disclosed. The method includes: displaying an image at a predefined original position in the 3D space near the display device; determining, by analyzing capacitive responses on the display device, whether a first interactive non-contact gesture has been performed by a user on the image at the original position; and, in response to the determination that the first interactive gesture has been performed by the user on the image, displaying at least a portion of the image as moved to a second position in the 3D space, and then returning to the original position after a predefined time. The method further includes determining, by analyzing capacitive responses on the display device, whether a second interactive non-contact gesture has been performed by a user on the image at the original position, and, in response to the determination that the second interactive non-contact gesture has been performed by the user, displaying at least a portion of the image as moved to a third position in the 3D space and returning to the original position after a predefined time. Attached Figure Description

[0010] To enable a more detailed understanding of the foregoing features of this disclosure, a more specific description of the disclosure, which has been briefly outlined above, can be obtained by referring to the embodiments, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings illustrate only some embodiments of this disclosure and should therefore not be construed as limiting its scope, as other equally effective embodiments are permissible.

[0011] Figure 1 This is an example system diagram according to one or more embodiments.

[0012] Figure 2AAn example gesture is shown, triggered by a touchscreen and display device according to one or more embodiments, to display a “genie” assistant icon in 3D space above the device.

[0013] Figure 2B The illustration shows a set of example (UI) buttons displayed in 3D space above an example touchscreen and display device, according to one or more embodiments, and then selecting one of the interface buttons "in the air".

[0014] Figure 2C The illustration depicts a user viewing an email projected in 3D space above an example touchscreen after the email icon has been selected, according to one or more embodiments.

[0015] Figure 2D This illustrates a user, according to one or more embodiments, who has read, as Figure 2C The email shown can be deleted by swiping in the air using the delete icon that appears in the air (after selecting the "Delete" icon).

[0016] Figure 3A An example system operating in traditional AR mode is shown.

[0017] Figure 3B The following is illustrated: operation in 3D AR mode according to one or more embodiments. Figure 3A An enhanced version of the example system.

[0018] Figure 3C The following diagram illustrates the situation when 3D AR mode is disabled, according to one or more embodiments. Figure 3B Example system.

[0019] Figure 4 An example capacitive response is shown, according to one or more embodiments, of a touchscreen device detecting the position of a user's finger in a non-contact grasping gesture.

[0020] Figure 5 It is according to one or more embodiments for detecting in Figure 4 The flowchart shows an example method for non-contact grasping and pulling gestures.

[0021] Figure 6 The illustration shows a user performing a non-contact squeeze gesture and a corresponding exemplary capacitive response to the squeeze gesture, according to one or more embodiments.

[0022] Figure 7 It is according to one or more embodiments for detecting in Figure 6 The flowchart shows an example method for a non-contact squeeze gesture.

[0023] Figure 8 This is a process flowchart of an example method according to one or more embodiments for engaging and disengaging 3D AR display functionality in response to various user command gestures.

[0024] Figure 9 This is a process flowchart of an example method for interacting with an aerial (aviation) display object according to one or more embodiments.

[0025] For ease of understanding, the same reference numerals are used wherever possible to denote the same elements common in the figures. Elements disclosed in one embodiment are expected to be advantageously used in other embodiments without specific description. Unless specifically indicated, the figures should not be construed as being drawn to scale. Furthermore, for clarity of presentation and explanation, the figures may be simplified and details or parts may be omitted. The figures and discussion are used to explain the principles discussed below, wherein the same reference numerals denote the same elements. Detailed Implementation

[0026] In one or more embodiments, non-contact gestures performed by a user in 3D space above the touchscreen of the combined display and touch-sensing device can be used to control the aerial display of images in 3D space. In one or more embodiments, gestures can be used to turn aerial image display functionality on or off for both AR applications and other use cases where direct or precise hand interaction with the touchscreen is inconvenient, undesirable, or impossible. In some embodiments, non-contact gestures can be used to turn mechanical or optical aerial display projection switches on the device on and off without requiring the user to make any hand contact with or even scan the device with their eyes.

[0027] The following description may use perspective-based descriptions such as top / bottom, in / out, above / below, etc. Such descriptions are used only to facilitate the discussion and are not intended to limit the application of the embodiments described herein to any particular orientation.

[0028] The following description may use the phrases "in one embodiment," "in one or more embodiments," or "in some embodiments," each of which may refer to one or more of the same or different embodiments. Furthermore, the terms "comprising," "including," "having," etc., as used with respect to embodiments of this disclosure are synonymous.

[0029] The terms “coupled with” and their derivatives, as well as “connected to” and their derivatives, may be used herein and included in the claims. “Coupled” or “connected” may mean one or more of the following: “Coupled” or “connected” may mean two or more elements in direct physical or electrical contact. However, “coupled” or “connected” may also mean two or more elements in indirect contact with each other, but still cooperating or interacting with each other, and may mean one or more other elements coupled or connected between the elements to be coupled or connected. The term “direct coupling” or “direct connection” may mean two or more elements in direct contact.

[0030] As used herein, including in the claims, the term "circuit" may refer to, be part of, or include: an application-specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or grouped) and / or memory (shared, dedicated, or grouped) executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality.

[0031] As used herein, including in the claims, "display device" can refer to a user device having both display and touchscreen functionality. As used herein, including in the claims, the term "display / touch panel" refers to the actual surface of the display device on which images are displayed and on which a user can hover and perform non-contact gestures. The panel may also sometimes be referred to herein as a "display screen," or, when the focus of the discussion is on the touch-sensing aspect of the display / touch panel, it may be referred to as a "touchscreen."

[0032] As used herein, including in the claims, the term "aerial display" refers to projecting a 2D image into space close to a display device, allowing a user to visualize the image without looking at the display device's screen. For example, an aerial display of an image by a display device can occur in front of, above, to any side of, or even behind the display device. "Aerial display mode," or, as sometimes used in the context emphasizing 3D aspects, "3D aerial display mode" of a display device, refers to an operating mode of the display device in which an aerial display occurs. It is conceivable from this disclosure that a user of a display device can turn the aerial display mode on and off in various ways, particularly, for example, without actually touching the display device's screen. In some examples, in an aerial display mode, an AR image is projected into 3D space above the display device, such that the image appears adjacent to or superimposed on a real-world object in 3D space as seen by the user at a distance from the display device. In other non-AR-based examples, user interface (UI) icons or images are projected into 3D space above the display device, allowing the user to interact with the display device when he or she would normally not touch it.

[0033] Figure 1 A schematic diagram of an example system 100 according to one or more embodiments is shown. Reference Figure 1 System 100 can be implemented on a display device, such as a smartphone, laptop computer, desktop computer, public information kiosk, or vehicle display. System 100 includes a camera 105, a touch IC 115, a processor 130, a display driver IC (DDIC) 132, a display 137, and a 3D imaging device 135. In one or more embodiments, the display 137 is configured to display images. Images may include real-time images acquired by the camera 115, or they may be images stored within or generated by the processor 130. Images stored or generated by the processor 130 may include user interface (UI) icons and menus, application icons, AR graphics, etc. Additionally, via a data network or cellular network connection (not shown), the display 137 may display images received from external sources, including video, such as photos or video clips downloaded by the user, or received from another person or application and pointed at the user.

[0034] The display 137 also includes touch sensing functionality in the form of a touch sensor array 138. The touch sensor array 138, together with the touch IC 115, comprises the touch sensing subsystem of the system 100. In one embodiment, the touch sensor array 138 includes an array of sensor electrodes 139 for performing capacitive sensing. In some embodiments, the sensor electrodes 139 of the touch sensor array 138 may include one or more common voltage electrodes. In some embodiments, the sensor electrodes 139 of the touch sensing array 138 may be integrated with electrodes for updating the display 137.

[0035] The sensor electrodes 139 of the touch sensor array 138 can have any shape, size, and / or orientation. For example, the sensor electrodes 139 can be arranged in a two-dimensional array, and each of the sensor electrodes 139 can be substantially rectangular in shape. In other embodiments, the sensor electrodes 139 can have other shapes. Furthermore, each of the sensor electrodes 139 can have the same shape and / or size. In other embodiments, at least one sensor electrode can have a different shape and / or size than another sensor electrode. In various embodiments, the sensor electrodes can be rhomboid, have interlacing fingers to increase field coupling, and / or have floating cutouts internally to reduce stray capacitance to nearby electrical conductors.

[0036] In one or more embodiments, some capacitive implementations of the touch sensor array 138 may utilize a "self-capacitance" (or "absolute capacitance") sensing method based on changes in capacitive coupling between the sensor electrodes 139 and an input object. In various embodiments, an input object near the sensor electrodes 139 (such as, for example, a finger, fist, palm, or set of fingers) alters the electric field near the sensor electrodes, thereby changing the measured capacitive coupling. In one implementation, the absolute capacitance sensing method operates by modulating the sensor electrodes 139 relative to a reference voltage (e.g., system ground) and by detecting the capacitive coupling between the sensor electrodes 139 and the input object.

[0037] In one or more embodiments, some capacitive implementations of the touch sensor array 138 may utilize “mutual capacitance” or “cross-capacitance” sensing methods based on changes in capacitive coupling between the sensor electrodes 139 themselves. In various embodiments, an input object near the sensor electrodes 139 alters the electric field between the sensor electrodes 139, thereby changing the measured capacitive coupling. In one implementation, the cross-capacitance sensing method operates by using a sensing signal 107 to drive some of the sensor electrodes 139 and capturing a received version of that signal (“result signal” 108) modified by the input object on the other sensor electrodes 139. Thus, in such an example implementation, the method detects capacitive coupling between one or more transmitter sensor electrodes (also referred to as “transmitter electrodes” or “transmitters”) and one or more receiver sensor electrodes (also referred to as “receiver electrodes” or “receivers”). The transmitter sensor electrodes may be modulated relative to a reference voltage (e.g., system ground) to transmit a transmitter signal. The receiver sensor electrodes may be kept substantially constant relative to a reference voltage, or modulated relative to the transmitter sensor electrodes to facilitate reception of the result signal. The resulting signal 108 may include one or more effects corresponding to one or more transmitter signals and / or one or more environmental interference sources (e.g., other electromagnetic signals). In one or more embodiments, the sensor electrode 139 may be a dedicated transmitter or receiver, or may be configured to both transmit and receive. In one or more embodiments, the touch sensor array 138, together with the touch IC 115, may use capacitive sensing to detect input objects that are close to (e.g., above) but not touching the display 137.

[0038] Continue to refer to Figure 1 In one or more embodiments, the touch IC 115 is configured to control the touch sensing array 138 to perform capacitive sensing. In one embodiment, the touch IC 115 includes a driver module (not shown) that may include a signal generator. In one or more embodiments, the driver module generates sensing signals 107 that drive the electrodes of the sensing array 138. In various embodiments, the touch IC 115 includes some or all of one or more integrated circuits (ICs) and / or other circuit components.

[0039] In some embodiments, the touch IC 115 also includes electronically readable instructions, such as firmware code, software code, and / or the like. In some embodiments, the components comprising the touch IC 115 are located together, for example, close to the touch sensor array 138. In other embodiments, the components of the touch IC 115 are physically separated from one or more components close to the touch sensor array 138 and one or more components elsewhere. For example, the system 100 may be physically integrated into a telephone or smartphone, and the touch IC 115 may include circuitry and firmware as part of the telephone's main processor. Furthermore, the touch IC 115 may be implemented in an automotive vehicle, and the touch IC 115 may include circuitry and firmware as part of one or more electronic control units (ECUs) of the vehicle.

[0040] In one or more embodiments, the touch IC 115 may be implemented as one or more modules (e.g., a driver module or a determination module) that process different functions of the touch IC 115. Each module may include circuitry, firmware, software, or a combination thereof as part of the touch IC 115. Different combinations of modules may be used in various embodiments. Example modules include a hardware operation module for operating hardware such as sensor electrodes and a display screen, a data processing module for processing data such as sensor signals and position information, and a reporting module for reporting information. Other example modules include a sensor operation module configured to operate one or more sensing elements to detect input, a recognition module configured to recognize gestures such as pattern-changing gestures, and a pattern-changing module for changing operating modes.

[0041] In one or more embodiments, the touch IC 115 may be combined with the DDIC 132, and the combined circuitry of the touch IC 115 and DDIC 132 may be configured for both display updates and input sensing, and may be, for example, referred to as including Touch and Display Driver Integration (TDDI) technology (not shown). In such embodiments, for example, the combined touch and display driver circuitry may be combined into a single IC, such as, for example, the TDDI 140 described below, such as... Figures 3A to 3C As shown. In this type of TDDI embodiment, the combined touch IC 115 and DDIC 132 circuits update the display 137 and control the input sensing of the touch sensor array 138.

[0042] In one or more embodiments, the ability of the touch IC 115 to detect the capacitive response of an object hovering over but not touching the display 137 via the touch sensor array 138 can be utilized to control the air display mode of the system 100 using the 3D imaging apparatus 135, and in particular, to control the display 137. As in Figure 1As shown, for example, touch IC 115 can detect a user's hovering action over an icon. Touch IC 115 can then notify processor 130 of this fact via signal path 116. Therefore, in one or more embodiments, touch IC 115 (or, in a TDDI embodiment, the touch portion integrating the TDDI circuitry) is responsible for acquiring hover or air gesture responses from sensor electrodes 139, abstracting them as gesture events, and reporting them to processor 130. In one or more embodiments, in response, processor 130 can provide display 137 with an icon image that the user can interact with via signal line 131, via DDIC 132 (or, in a TDDI embodiment, via the display driver portion of the combined touch and display ICs).

[0043] Continue to refer to Figure 1 Camera 105 is configured to detect the positions of one or more real-world objects near system 100 and report these positions to processor 130 via signal path 106. This functionality is used in an example aerial display mode of system 100. Thus, for example, in such an aerial display mode, processor 130 can generate AR graphics to enhance real-world images of such objects, send them to display 137 (via DDIC 132 as described above) to display them to the user in 3D space above display 137, such that they are at or appear to be at the precise location of the real-world object. In this way, the user sees an enhanced version of the real-world object enhanced by AR graphics. See below for further details. Figure 3B Further details and examples describing this functionality are provided. Figure 3B This example shows a 3D AR mode for a display device.

[0044] The following description Figure 2A An example user interaction with icons on a display device, according to one or more embodiments, is shown to enter an over-the-air display mode of the display device.

[0045] therefore, Figure 2A An example gesture, triggered by a touchscreen-equipped display device, is shown to display an example "Sprite" 232 icon in 3D space above the device, according to one or more embodiments. In one or more embodiments, the example gesture is a "non-contact" gesture, meaning that the user does not need to touch the display device in any way to perform it. Non-contact gestures thus provide a way for the user to control the display device in a context where touching the display device would be undesirable for him or her for various reasons. Reference Figure 2AThe diagram shows three panels, each representing a different point in the time series, starting with the leftmost panel (panel 210) and ending with the rightmost panel, panel 230. Figure 2A Each panel illustrates an example display 137 of the example display device 129. In panel 210, at the first point in the sequence, a "sprite" icon is displayed on display 137, which in one embodiment serves as an assistant icon to guide the user to activate the device's over-the-air display mode. In panel 210, as shown, there is no user interaction with display 137.

[0046] Following the scenario depicted in panel 210, in panel 220, at a later point in time, the user's hand 201 hovers over display 137, and specifically over sprite icon 211, without touching display 137 or display device 129. In response, the touch sensing system of display device 129 detects the hovering user's hand 201. If the user then performs a first non-contact gesture, in response, as shown in panel 230, the system displays sprite icon 231 on display 137 and above it in preview mode of the air display mode. In one or more embodiments, displaying icon 232 in preview mode before entering full air display mode gives the user an opportunity to confirm that he or she wishes to select the full air display mode of the example display device. In one or more embodiments, entering air display mode can consume power, so requiring the user to perform a two-stage process to enter it is useful. In one or more embodiments, if the user performs a first non-contact gesture after hovering over display 137, as shown in panel 220, the preview air display mode can be entered. Reference Figure 2A Between the situations depicted in panels 220 and 230, the user only performed such a first non-contact gesture (not shown).

[0047] Continue to refer to Figure 2A After the user performs a contactless gesture (as indicated, the first contactless gesture is performed in...), Figure 2A (Not shown in the image), as shown in panel 230, a sprite or assistant icon 231 is now displayed on display 137 and is also projected into 3D space above display device 129 in preview mode 232. In preview mode, the projected image of sprite icon 232 is seen at a first relatively low height above display 137. At this time, in one or more embodiments, if the user wishes to enter the all-air display mode, he or she will perform a second non-contact gesture. Figure 2AAs shown at panel 230, the second non-contact gesture may include grasping the projection of a sprite icon 232 in 3D space above the display device 129, as illustrated, and then pulling it upwards. As described below, in some embodiments, this is known as a "pull" gesture. In one or more embodiments, following this second non-contact gesture (e.g., a pull gesture), the display device enters an all-air display mode. In the all-air display mode ( Figure 2A In a 3D space above the display device (not shown), icons, graphics, or other images can be projected at a second, relatively high height, making it easier for users to interact with them.

[0048] Once in full display mode, users can interact directly with the UI graphics and control various functions of the display device using only contactless gestures. Examples of such interactions will be described below. Figures 2B to 2D As shown in the image.

[0049] Figure 2B The illustration depicts a user's hand interacting with example UI graphic buttons displayed in 3D space above an example touchscreen and display device, according to one or more embodiments. As shown, four UI graphic buttons 172 are displayed on display 137, and four identical UI graphic buttons 173 are also projected upwards into the air above display 137. The first graphic 168, 169 in each set is an envelope icon for launching an email program. In one or more embodiments, by interacting with the projected set of graphics 173 using a finger 135 in space, a user can have the same interactive functionality as if he or she were physically touching an actual display screen, where the icons are displayed on display 137.

[0050] Figure 2C The illustration depicts a scenario where a user, having selected email icon 169 according to one or more embodiments, is reading an actual email 176 that has now been projected into 3D space above example display 137. As shown, display 137 displays email program icon 168, and a displayed version of the actual email 175 from Jim asks the user if it has arrived. As noted, the email is displayed on display 137 and is also projected into 3D space above aerial display 176.

[0051] Figure 2D This shows that the user has read, such as Figure 2CThe projected version of email 176 shown is now being scanned in the air above display 137 with the user's finger 135 to delete it after the "delete" icon 179 (selection not shown) has been selected. Email 175 and the delete icon 178 are also displayed on display 137 in a "land" version, but the user does not even need to look at those images on display 137 to interact with the email, as shown. This illustrates the significant benefits of the air display mode according to one or more embodiments. Thus, in the scanning motion 177, the user scans his or her finger 135 from right to left in the space above display 137, where the air display version 176 of the email is projected. When the user performs the delete scan 177, the letters of the email are erased sequentially, as shown. As the text of email 175 is displayed on display 137, when the user scans 177 over the projected air image 176 of the email, the ("land") text of email 175 displayed on display 137 is also erased in the exact same manner as the air-displayed version. Therefore, in versions 175 and 176 of the email, the words “Jim” and “you have” are not shown, but have been “pushed” to the left by the sweeping motion 177.

[0052] At this point, the attached document has been consulted. Figures 2A to 2D This describes an example of user interaction with UI icons in the over-the-air display mode of an example display device. See below for further details. Figure 3B and Figure 3C A 3D AR mode of an example display device according to one or more embodiments is described. To facilitate this description, reference is first made to... Figure 3A Describes the standard AR mode.

[0053] Figure 3A This describes an example system 300A that can be placed in a display device operating in a conventional AR mode. As described below, in a conventional AR mode, the display device adds a graphical image to an image of a real-world object to create an enhanced version of that real-world object. However, this enhanced world must be viewed on a display screen. (Reference) Figure 3A It shows the relationship with Figure 1 The example system 300A is similar to system 100. Example system 300A can be provided, for example, in a car, particularly in a navigation or infotainment system. Example system 300A can, for example, have an AR mode, in which images of real-world objects in or near the road on which the vehicle is traveling are displayed on an in-vehicle display 145 together with AR graphics 170 that add information or interpretation to the real-world image 160.

[0054] refer to Figure 3AStarting at the far left of the figure, system 300A includes a camera 110 that captures raw images 161 of real-world objects 160, such as, for example, pedestrians. The raw images are fed to a central processing unit (CPU) 130, which processes the raw images 161 and adds AR graphics 170 to them. For example, as shown, the AR graphics are labels “pedestrian” placed adjacent to the images of the pedestrians. The combined image 161 with the added graphics 170 is fed to TDDI circuitry 140, which causes the combined in-vehicle display / touch panel 145 to display the now enhanced and processed image. Whenever an object is sensed near the display / touch panel, the display / touch panel 145 also provides a proximity raw signal 146, such as, for example, a user’s hand in one of several postures. As described below, the proximity raw signal 146 can be used to alert TDDI 140 to analyze the capacitive response of such nearby objects and identify one or more of them as user commands.

[0055] Continue to refer to Figure 3A Below the schematic diagram of system 300A, an in-vehicle display 144 of the display / touch panel 145 is also shown. On the in-vehicle display 144, the user / driver sees an enhanced image 162, which includes an image 161 of a pedestrian, such as an AR graphic 170, with the word "pedestrian" written in a rectangular field. Thus, as shown, the user can see a real-world image of the pedestrian 160 directly through his windshield, and can also see an enhanced image 161 of the pedestrian with an identifier representing "pedestrian" 170 on the in-vehicle display 145. Thus, in conventional AR mode, both the image of the real-world object and the associated AR graphics generated by the application are displayed with a 2D display but only on a 2D display. This forces the user to look at the 2D display 144 of the display / touch panel 145 to see the enhanced image 162, but then frequently shift his or her gaze to the actual real-world object 160 outside the vehicle on the actual road. This is both distracting and uncomfortable, and can be dangerous. In one or more embodiments, this problem is addressed as described below.

[0056] Figure 3B and Figure 3C The enabled and disabled states according to one or more embodiments are shown respectively. Figure 3A An enhanced version of the example system. Reference Figure 3BThe diagram illustrates system 300B. System 300B is configured to operate in a 3D AR mode, wherein it is configured to display AR graphics in the air at object coordinates in 3D space above display 145, each associated with a real-world object. In doing so, the user sees the AR graphics as if they were on or near an actual real-world object. Thus, for example, a user of a cooking instruction app running on a display device including system 300B would see an AR label or other graphic displayed next to, for example, an onion he or she is slicing on a kitchen surface, indicating where the user should cut the onion with a chef's knife, what type of stroke to use, and how thin the slice should be. It can also provide feedback on the cut after the user has finished slicing the onion. In one or more embodiments, this display of AR graphics at precise object coordinates can typically be achieved, for example, by fixing the relative positioning of the user's eyes and the position and orientation of the display device (e.g., the user places the display device where he or she will use it during the application and also stands in the location where he or she will use the application), and then calibrates it before displaying the AR graphics.

[0057] In the example in-vehicle embodiment, such as in Figure 3B As illustrated in the example, where the position of the driver's seat and the positions of the display device and its camera are fixed, object coordinates are somewhat easier to estimate, and all that is needed is some initial calibration to account for variations in user height and the gestures they use while seated in the driver's seat. In one or more embodiments, this can be performed once for each user of the vehicle and does not need to be repeated. In one or more embodiments, engagement and disengagement from 3D AR modes are controlled by non-contact gestures sensed and interpreted by the TDDI circuit 140, as described in more detail below. Therefore, system 300B has superior performance compared to... Figure 3A Some enhancements to the standard system 300A shown.

[0058] Because most of the components in System 300B are the same as those in System 300A, and have similar functions unless otherwise stated, they do not need to be described again, or only briefly. See also: Figure 3BA camera 110 is present, which is used to capture a real-world image 160, as described above. Similar to the case of system 300A, the raw image is fed to a central processing unit (CPU) 130, which processes both the raw image 160 and adds AR graphics 170 (shown here as the identifier "pedestrian," which in this example is the driver of the vehicle) to be displayed, for example, on the windshield of the vehicle, so that the user / driver appears to be outside the vehicle, in the same position as the actual object (pedestrian 160) outside the vehicle. As described above, for an in-vehicle display application that has been calibrated for a given user, the application can then estimate the height at which the AR graphics should be displayed. Part of the CPU 130's processing of the raw image of object 160 is determining the coordinates of object 160 in a common 3D space 165 shared by the volume of both object 160 and the display / touch panel 145 above or in front of it (depending on how it is mounted).

[0059] Continue to refer to Figure 3B The CPU 130 provides AR graphics to the TDDI circuit 140, which feeds them to the display / touch panel 145 using a display driver. Additionally, the TDDI 140 is configured to sense non-contact gestures from the user of the system formed over the surface of the display / touch panel 145. For example, these non-contact gestures can be used to preview, turn on, and off air display functionality. Figure 3B For the purpose of this study, let's assume, for example, that a user of a system provided in a vehicle has performed a contactless "grab" gesture, followed by a contactless "pull" gesture, which is detected by TDDI 140, as shown in 141. These contactless gestures, explained in detail below, first place the air display function of system 300B in preview mode, and then in full operation mode, respectively. Once this occurs, TDDI 140 sends a signal to CPU 130, which in turn sends both an enable signal 131 to camera 110 and an enable signal 133 to a mechanical / optical switch configured to project an air image into 3D space 165. In summary, these enable signals 131, 133 enable camera 110 to acquire the coordinates of any identified object in 3D space, and the mechanical / optical switch 135 controls the projection of AR graphics 170 generated by CPU 130 into a 3D imaging apparatus (not shown) in 3D space 165, as illustrated.

[0060] Therefore, as Figure 3BAs shown in the lower right, because the AR graphics 170 are all displayed on the display / touch panel 144 and also projected onto the air display via an optical medium such as a mechanical / optical switch, as described above, the user sees object 160 in the real world along with its spatially displayed AR graphics 171, but no image of the real-world object 160 is shown on the display 144. Therefore, as long as the system 300B is in air display mode, the vehicle user / driver does not need to look at the display 144 at any time. As described in detail below, in one or more embodiments, in order to turn off the air display function, the user can perform a third non-contact gesture, such as a "squeeze".

[0061] Although System 300B shows an in-vehicle example, there are many other examples of situations where operating a display / touch panel device in an over-the-air display mode would benefit. These methods include, for example, browsing a cooking recipe app with dirty or wet hands, or browsing instruction steps using a woodworking app while wearing gloves, making it impossible to physically touch the touch panel of the display device; or, for example, performing a medical procedure or operation in which one's eyes must be on the patient and the doctor or surgeon is also wearing gloves.

[0062] As referenced above Figure 3B As described above, once TDDI 140 detects that the user has performed a contactless grasping and pulling gesture, CPU 130 sends enable signals 131 and 133. Similarly, once TDDI 140 detects that the user has performed, for example, a contactless squeeze gesture, CPU 130 sends disable signals to camera 110 and mechanical / optical switch 135, and these cease performing over-the-air display functionality. (See below for further details.) Figure 3C Describe this.

[0063] Figure 3C The following diagram illustrates the situation when the over-the-air display mode is disabled, according to one or more embodiments. Figure 3B The example system operates as follows. System 300B is disabled by sending a disable signal 131 to camera 110 via CPU 130, and similarly by sending a disable signal 133 to mechanical / optical switch 135. In one or more embodiments, a definitive disable signal may be sent, or (e.g.) an enable signal sent by CPU 130 may be absent (such as...). Figure 3B (As shown in the image) can also be operated as a "disable" signal. Therefore, as... Figure 3CAs shown, because the 3D AR mode has been disabled, the mechanical / optical switch has turned off the air image display, and the UI graphics 172 generated by the CPU 130 are simply displayed on the touch / display 145, as shown. The user sees object 160 in their view of 3D space 165, but he or she cannot see graphics or AR enhancements in 3D space 165. This occurs as long as no grabbing or pulling user non-contact gesture is detected, as shown in 142. However, if the user decides to re-enable the air display mode, such a gesture must be made, and once detected, the example system 330B returns to its configuration, as shown in Figure 3B As depicted in the text.

[0064] Figure 4 Example capacitive responses of a user's finger at different example positions in a touchscreen of an example device, according to one or more embodiments, are illustrated in relation to the detection of a non-contact grasping gesture followed by a pulling gesture. It should be understood that the non-contact gestures described in this disclosure (including...) Figure 4 and Figure 6 The examples shown are merely illustrative, and any set of mutually identifiable contactless gestures can be mapped to a corresponding set of control commands for the example touch and display system. (See above references.) Figure 3B As noted, in some embodiments, a two-stage process, referred to as a "grab" non-contact gesture followed by a "pull" non-contact gesture, can be used to engage the example system's air display mode by first entering a preview of the air display mode after the "grab" gesture, and then entering the example system's full air display mode after a subsequent "pull" gesture performed within a predetermined time. In one or more embodiments, capacitive proximity responses are detected and processed on TDDI circuitry (e.g., a TDDI chip) or on a touch controller of a touch-sensing display device in other embodiments. Once a corresponding non-contact gesture of "grab" or "pull" has occurred, it is reported to a host, which then enables a mechanical / optical switch to perform an air display projection.

[0065] According to one or more embodiments, in Figure 4 The document illustrates details of exemplary "grab" and "pull" gestures and how they are detected and reported based on their respective finger configurations. (Reference) Figure 4At 410, a user's hand having a plurality of fingers extending from each other is first lowered downward toward the touch screen, as indicated by arrow 411, ending at a height as shown at 413. For example, the fingers may be a thumb and an index finger. Once the user's fingers involved in the gesture are positioned as shown at 413, the user's fingers are now close enough to the touch screen to generate a capacitive response 414. As shown, the capacitive response 414 for two fingers in configuration 413 is viewed as two or more humps, each hump representing a given maximum signal amplitude and corresponding to one of the extended fingers. In one or more embodiments, the relative peak-to-peak distances (ΔX, ΔY) of the humps, their heights and / or their end-to-end width (W) may be stored at an initial encounter, such as shown at 413, designated t=T0 405. In one or more subsequent time instances, where t<T0+Td 406, for a predefined time window Td, the fingers may have a configuration as shown at 415, and may have a capacitive response 416. A comparison may then be made of the metrics of the two capacitive responses at, for example, the two data points. If upon comparing the proximity of the peaks of the humps at t=T0 and t=some subsequent time <T0+Td, it is determined that the two humps of the capacitive response gradually converge to the same spatial position and eventually become a single object entity, or at least become closer, for example, by measurement, reducing (ΔX, ΔY) or W below a certain threshold, a grab gesture may be reported to the system host. The grab gesture indicates that the user has pinched two or more fingers together, as shown by the finger configuration at 416.

[0066] Although in Figure 4 only a two-dimensional (2D) cross-section of two capacitive responses 414 and 416 is shown, it should be understood that, generally, Figure 4 the peaks of the capacitive response shown therein actually behave as a 3D "hill" with a peak at the center, which is surrounded by a set of nested annular curves with increasingly lower amplitudes as their distance from the central peak. This is because a finger, being a 3D object hovering above a 2D sensor array, has a 3D capacitive response. Therefore, the actual distance between the peaks in configuration 414 is , and similarly, the actual distance between the peaks in configuration 416 is , wherein the distance between two peaks may be a line segment in the X-Y plane. Similarly, as shown in Figure 4 , W refers to the distance between the outer edge of one such "hill" and the outer edge of another "hill" of the capacitive response along a line in the X-Y plane passing through their respective centers.

[0067] In one or more embodiments, in response to the host receiving a grasping gesture instruction, it may then activate a mechanical / optical switch, such as, for example... Figure 3B and 3C The system 300B uses a mechanical / optical switch 135 to display one or more images in the air in preview mode. In one or more embodiments, in the preview mode of the display device, the image is projected into a 3D space above the display screen, but at a relatively low height. For example, the height of the projected image above the display screen in preview mode may be a portion of the full height over which the system can project an image. In some embodiments, the image projected into the 3D space may be as follows: Figure 2A The image shown is an assistant or sprite icon used to guide the user from the preview aerial display mode to the full aerial display mode. Once the latter occurs, the user can interact with the actual AR or UI graphics displayed in the air, depending on which task or application the user has selected.

[0068] Continue to refer to Figure 4 In one or more embodiments, when a pull gesture by a user is detected by, for example, a TDDI chip or touch controller, a full-air display mode is enabled, where the image is projected at its full height above the display screen, provided the pull gesture is made within a predefined time Td following the initial grasp gesture. An example pull gesture is shown by arrow 419, where the user holds his or her finger in substantially the same configuration as shown at 415 and pulls or raises his or her finger away from the display screen, as indicated by upward-pointing arrow 417. Regarding the capacitive response graph of the pull operation, it maintains a similar peak-to-peak distance as it has at the end of the grasp gesture, since the peaks do not move relative to each other. However, W will decrease during the pull gesture because the amplitude of the capacitive response decreases across the entire response graph as the distance from the object (the user's finger) to the display screen increases, and thus the end of the capacitive response graph drops to zero. Therefore, in one or more embodiments, a pull gesture is detected by a gradual decrease in the overall amplitude of the capacitive response, while the shape of the capacitive response remains substantially the same, but its width W decreases. This two-stage action of grabbing and pulling allows for previewing or confirming the user's aerial projection and also prevents any erroneous activation, such as that which might be caused, for example, by an accidentally waving hand on the display. In some embodiments, to perform a pull gesture, the user can pull an icon or other image at the location displayed in preview mode in response to a grab gesture, such as, for example, on Figure 2A As shown, the user's hand 201 pulls the sprite icon 232 upwards. Pulling the projected image gives the user an easy landmark to pull on.

[0069] Figure 5is an example method 500 for detecting and responding to Figure 4 each of the non-contact grasping and pulling gestures shown in . a process flowchart of the example method 500. Method 500 includes blocks 501 to 560. In alternative embodiments, method 500 may include more or fewer blocks. Method 500 begins at block 501, where an air display mode of a display device is disabled. For example, this is the situation described above Figure 3C shown in . Method 500 proceeds from block 501 to decision block 505, where it is determined whether at least two objects are hovering over the display screen of the display device. For example, there may be two fingers hovering over the touchscreen, as shown Figure 4 at 413 of . If the answer is "yes" at decision block 505 and at least two hovering objects are detected, method 500 proceeds to block 515, where the X and Y coordinates of the at least two objects at an initial time t=T0 are updated and stored, for example X1, and X2, . On the other hand, if the answer at decision block 505 is "no", method 500 returns to decision block 505 and continues checking for the at least two hovering objects, as shown.

[0070] Method 500 proceeds from block 515 to decision block 520, where it is determined whether t<T0+Td. This inquiry is essentially a timeout test, in which it is determined whether a time greater than Td has elapsed since the at least two hovering objects were detected at t=T0. If the answer is "yes" at decision block 520, method 500 proceeds to block 525, where the X and Y coordinates of the at least two objects at this second time point (where T0<t<T0+Td) are stored, for example X1, and X2, for comparison with the coordinates obtained and previously stored at block 515. On the other hand, if the answer at decision block 520 is "no", method 500 returns to decision block 505 and continues checking for the at least two hovering objects, as shown. For ease of description, it is assumed that there are only two objects, such as, for example, Figure 4 two fingers shown in .

[0071] Method 500 proceeds from block 525 to decision block 530, where it is determined whether the peak-to-peak distance of the capacitive responses of the two objects has decreased by at least a first threshold D1. Thus, for example, whether: .

[0072] If a "No" response is returned at query box 530, method 500 returns to query box 505 and, as noted, continues to examine at least two hovering objects. However, if a "Yes" response is returned at query box 530, then the two objects have become closer than the threshold D1, and this has been done within the time interval Td. Therefore, a grasping gesture has been identified, and method 500 proceeds to box 535, where, for example, the grasping gesture is reported to the host computer, allowing a preview of the 3D aerial display mode to be provided to the user. The preview may include, for example, projecting an image of a sprite or other icon at a low height above the display screen, as described above. In some embodiments, the sprite or other icon is projected at the same height as when the grasping gesture was performed, whereby the user's finger is then positioned.

[0073] Continue to refer to Figure 5 Method 500 proceeds from box 535 to query box 540, where it is determined whether an object (e.g., a finger) has been lifted to a relatively high height above the display screen. For example, in some embodiments using capacitive sensing, the maximum detection height of the object can be 5 cm above the display screen. Therefore, in such an embodiment, a sprite icon can be displayed first at, for example, 3 cm after a grasping gesture, and then, if the finger is seen to disappear (corresponding to the user lifting it to a level above 5 cm), a pull gesture should be triggered. Therefore, if "yes" is returned at query box 540, method 500 proceeds to box 560, where the pull gesture is reported to the host computer, and the host computer initiates a 3D air display mode. Method 500 then terminates at box 560.

[0074] On the other hand, if the response at query box 540 is "no", and therefore no lift gesture has been detected, the user may still maintain a grab gesture (e.g., there may still be a similar...). Figure 4 If the capacitive response shown at 416 is a capacitive response, then method 500 proceeds to query box 541 to test whether sufficient time has elapsed to make a new determination. For example, in one or more embodiments, the display device may wait until a new touch sensing frame has begun in order to proceed. Therefore, at query box 541, it is determined whether t > T0 + Td. If "No", then method 500 loops back to query box 541 until the required time has actually elapsed. If "Yes" is true at query box 541, then method 500 proceeds to box 545, where the X and Y coordinates are updated for t > T0 + Td in the same manner as described above for box 525, and method 500 proceeds to query box 550, where it is determined whether the peak-to-peak distance between the two objects has decreased overall by at least a second threshold D2, which is greater than D1. Therefore, for example, whether .

[0075] If "yes" is returned at prompt 550, the two objects have become even closer and are now within a distance D2 of each other, so the user has not yet abandoned the grab gesture. Method 500 then proceeds again to prompt 540 to test for a pull gesture, and if "no," method 500 can continue looping through prompts 541, 545, 550, and 540 until a pull is detected at prompt 540, or "no" is returned at prompt 550. If the latter occurs, the user's finger, or whatever other object is being used, is not getting closer, and method 500 proceeds to prompt 555, where a "non-grab" gesture is reported, meaning the user has abandoned the grab gesture. This allows the host computer to collapse the icon already displayed above the screen at preview mode height, and method 500 returns to prompt 505 to check the two objects hovering again, as described above. Therefore, the display device remains in the air-display preview mode until a "No" is received at query box 550 (or, of course, a "Yes" is returned at query box 540), and the icon or image is continuously projected at the preview height. In an alternative embodiment, a timer can be set, for example, by implementing another query box to the left of query box 541, to test how much time has elapsed since T0, and if t > T0 + N * Td, where N is a predefined integer, then method 500 terminates even if the grab gesture has not been abandoned.

[0076] Figure 6 An example of a user performing a non-contact squeeze gesture and a corresponding example of a capacitive response to the squeeze gesture are illustrated according to one or more embodiments. References Figure 6 At 605, a user's palm is shown lowered towards the display in a substantially horizontal position, and similarly, at 610, a user's fist is shown also lowered towards the display in a horizontal position. As mentioned above, the closer the object is to the touch sensing system, the higher the amplitude of the capacitive response. Therefore, as in Figure 6 As shown, it is possible to define with Figure 4 The third type of non-contact gesture, distinct from the grasping and pulling gestures shown, and in one or more embodiments, can be used to signal to the display device that the user wants to terminate the display device's 3D air display mode. To eliminate the ambiguity of normal fingertip gestures that can be used for interaction in air display mode, a palm or fist is used for squeezing gestures, as each of these objects has a larger and flatter signal profile.

[0077] Continue to refer to Figure 6Both contactless gesture options shown have a capacitive response 620 with a width W 615. As the user's hand descends, the amplitude of the capacitive response 620 increases, and the width W of the capacitive response increases because more sensors on the touch / display detect the presence of the hand (not just the sensor directly below the hand). In one or more embodiments, this increased width W, along with the increased total amplitude of the capacitive response, can be detected as a squeeze-to-contact gesture. Figure 6 As shown, in one or more embodiments, W can be measured at a point above the bottom of the response graph to avoid noise.

[0078] Despite Figure 6 The image shows two example hand gestures, 605 and 610, but various other gestures can also be used for squeeze gestures. For example, the Italian finger wallet gesture, where the palm is back-down and the fingers touch at their respective tips, is sometimes performed when saying "ma che vuoi" (you want something), and will also have a similar capacitive response, and can also be used as a hand gesture for squeeze gestures when lowered on the display.

[0079] Figure 7 It is according to one or more embodiments for detecting in Figure 6 The flowchart illustrates an example method 700 for a non-contact squeeze gesture. Method 700 includes blocks 705 through 745. In alternative embodiments, method 700 may have more or fewer blocks. Method 700 begins at block 705, where an air display mode has previously been enabled for the display device. Therefore, Figure 7 Assuming that... Figure 3B The system described in the text.

[0080] Method 700 proceeds from box 705 to query box 710, where it is determined whether an object is hovering over the display screen of a display device. For example, in Figure 6 In either of the two poses shown, a user's hand may be hovering over the display device. If the response at query box 710 is "yes," then method 700 proceeds to box 715, where the capacitive response W of the object at this initial time t = T0 is updated and stored. On the other hand, if the response at query box 710 is "no," then method 700 returns to query box 710 and continues to check the hovering object, as shown.

[0081] Method 700 proceeds from box 715 to query box 720, where it determines whether the capacitive response W detected at t = T0 satisfies a minimum threshold W1. The minimum threshold is to ensure that there is sufficient capacitive response at t = T0 to serve as a basis for comparison. If the user's hand is too high above the display, resulting in a very small amplitude and therefore W1 being less than the threshold W1, the return at query box 720 will be "No". If the return at query box 720 is "Yes", method 700 proceeds to query box 725, where it determines whether t > T0 + Td to see if enough time has elapsed for the user to have performed the squeeze gesture. (See above reference.) Figure 5 The specified Td is also a sufficiently long time interval so that the next measurement of W appears in a new capacitive sensing frame. If the response at query box 725 is "yes", then method 700 proceeds to box 730, where the capacitive response W of the object at this second time t>T0+Td is updated and stored. On the other hand, if the response at query box 725 is "no", then method 700 loops back to query box 725 until a time such as t>T0+Td is reached.

[0082] Method 700 proceeds from box 730 to query box 740, where it determines whether the W of the capacitive response detected at t>T0+Td is now greater than a second threshold W2, where W2>W1. If so, it means that the user's hand has sufficiently descended to allow the inference of an intention to perform a squeeze gesture, and is not a normal slight upward or downward movement, which could occur if the user only intended to keep his or her hand in a fixed position. If the response at query box 740 is "yes," method 700 proceeds to box 745, where the squeeze gesture is reported to the host computer, and the air display mode ends. Method 700 then terminates. On the other hand, if the response at query box 714 is "no," then a squeeze gesture has not been performed, and method 700 returns to query box 710 and continues to examine the hovering object, as shown in the figure.

[0083] Figure 8 A method for engaging and disengaging air display functionality in response to various user command gestures, according to one or more embodiments, is illustrated. For example, the electronic device may be a combination of a display and a sensing device, such as a display device and a sensing device including, for example, TDDI technology, as described above. For instance, the display device may include… Figure 3B and Figure 3C System 300B.

[0084] Method 800 includes frames 810 through 860. In alternative embodiments, method 800 may have more or fewer frames. Method 800 begins at frame 810, where a first non-contact gesture is detected over a touchscreen-enabled display device. For example, the gesture may be as shown in... Figure 2A The hover gesture shown, or for example, as Figure 4 The "grab" gesture 415 shown.

[0085] Method 800 proceeds from box 810 to box 820, wherein, in response to the detected first gesture, a preview 2D image is displayed in 3D space above the display device. For example, the preview 2D image could be a "sprite" or user assistant icon 231, such as... Figure 2A As shown, or for example, some equivalent assistant icons, which could be a set of user interface icons 173, including email icon 169, such as Figure 2B As shown. In some embodiments, the preview 2D image may be displayed in a manner smaller than the "full service" image. For example, Figure 2A The sprite icon 231 can be projected above the monitor at a shorter distance than it would be if the user indicated that he or she wished to use the air display mode.

[0086] Method 800 proceeds from block 820 to block 830, wherein a second non-contact gesture is detected above the display device within a predefined time period. This second gesture has a capacitive response different from the first non-contact gesture. In an embodiment, as described above, the first gesture is used to trigger the display of a preliminary or preview image, indicating to the user that air display functionality is available. In an embodiment, when the user performs the second gesture, it is interpreted as confirmation that the user wants to use the air display functionality of the display device. Therefore, the first and second gestures need to be distinguished from each other by the display device, and thus, in this embodiment, they have different capacitive responses.

[0087] For example, the second gesture could be a "pull" motion performed on the displayed preview 2D image when projected into 3D space, such as, for example, the user pulling upwards on the sprite icon 231 or an equivalent assistant icon, as in... Figure 2A As shown in panel 230. Depending on the situation, sprite or assistant icons are used to provide the user with visual feedback effects of a launch screen nature to notify the user that air display projection is enabled or disabled. Once the user is fully engaged in air display mode, various other icons may be projected, depending on the application selected by the user.

[0088] Method 800 proceeds from box 830 to box 840, wherein, in response to the detected second gesture, the image for preview is now displayed entirely in 3D space above the input device. For example, the sprite icon 231 is now displayed at a height higher than it was during the preview.

[0089] Method 800 proceeds from block 840 to block 850, wherein, when the display device is in an air display mode, for the space above the third non-contact gesture monitoring input device, the third non-contact gesture has a capacitive response different from either the first or second non-contact gesture. For example, the third non-contact hand gesture could be, for example, in... Figure 6 The squeeze gesture shown has one of two example hand configurations, or another configuration with an equivalent capacitive response.

[0090] Method 800 proceeds from box 850 to box 860, where, in response to the detection of a third non-contact gesture, the air display mode of the display device is terminated, and method 800 ends. In some embodiments, upon detection of a squeeze gesture, whichever icon is projected is simply folded, and the user must subsequently interact with them on the display itself. In an alternative embodiment, upon detection of a squeeze gesture, not only UI or AR, but depending on the situation, icons and images are no longer projected upwards; sprite or assistant icons may appear and be displayed as if descending into the screen to visually confirm to the user that a squeeze has occurred. In some embodiments, a "squeeze" or termination sound may also be played to provide the user with additional audible cues. Indeed, in such embodiments, audible cues may be played to the user to indicate both preview mode and full air display mode.

[0091] Figure 9 This is a process flowchart of an example method for interacting with a user performing contactless gestures, according to one or more embodiments. Although Figure 8 Example method 800 illustrates the use of contactless gestures to preview, enter, and stop the device's air display mode. However, method 900 also provides additional contactless gestures that can be used while in air display mode to interact with one or more applications via, for example, the same icon used to enter air display mode (as described above, a sprite or assistant icon). Therefore, method 900 continuously displays the icon, such as the sprite or assistant icon, until the display device terminates the air display mode. According to one or more embodiments, method 900 can be performed by a display device equipped with air display functionality. For example, the electronic device may be a combination of display and sensing devices, such as a display device and sensing device including, for example, TDDI technology, as described above. For example, the display device may include the features described above. Figure 3B System 300B.

[0092] Method 900 includes frames 910 through 960. In alternative embodiments, method 900 may have more or fewer frames. Method 900 begins at frame 910, where the display device displays a black screen. Method 900 proceeds from frame 910 to an interrogation frame 915, where it is determined whether a grasping gesture has been detected in 3D space above the display device. If "yes" is returned at interrogation frame 915, then method 900 proceeds to frame 920, where an icon is displayed at the bottom of 3D space above the display device to indicate a preview of the air display mode. For example, the icon could be... Figure 2A The sprite icon 231. On the other hand, if the response at query box 915 is "no", the grab gesture has not been performed, and method 900 returns to box 910 and continues to display a black screen.

[0093] Method 900 proceeds from box 920 to query box 925, where it is determined whether a pull gesture has been detected in the 3D space above the display device within a predefined time. If the response at query box 925 is "yes," then method 900 proceeds to box 930, where the icon previously displayed at the bottom of the 3D space above the display device now rises to the top portion of the 3D space to indicate that an air display mode has been entered. This upper position of the icon is referred to as its "home position." On the other hand, if the response to query box 925 is "no," then no pull gesture has been executed in time, and method 900 proceeds to box 960, where the icon disappears, and method 900 terminates. Alternatively (not shown), method 900 may not terminate at box 960, but instead return to box 910 and wait for the next grab gesture.

[0094] Method 900 proceeds from box 930 to query box 935, where it is determined whether a "tipleft" gesture has been detected. The tipleft gesture is a fourth non-contact gesture used by the user to control or command functionality while in full-air display mode. The tipleft gesture is applied by the user to a sprite or assistant icon used for previewing and entering air display mode. In one or more embodiments, the tipleft gesture can be mapped to any desired functionality of any application on the display device. If "yes" is returned at query box 935, then method 900 proceeds to box 940, where the icon is caused to tip left and then bounce back to its original position at the top portion of 3D space. Simultaneously, the functionality mapped to the tipleft is performed by the display device, and method 900 proceeds to query box 945. In some embodiments, the icon rotates only while in its original position to indicate a tipleft. In other embodiments, for example, in a more illustrative version, the icon may all translate to the left as it rotates. Other example movements may also be performed. However, if "No" is returned at query box 935, then method 900 proceeds directly to query box 945, where it is determined whether a "tilt to the right" gesture has been detected. A tilt to the right gesture is the fifth non-contact gesture used by the user to control or command functions while in full-air display mode. As with a tilt to the left gesture, the user applies a tilt to the right gesture to the sprite or assistant icon used to preview and enter air display mode. In one or more embodiments, a tilt to the right gesture can be mapped to any desired functionality of the display device.

[0095] If the response at query box 945 is "Yes", then method 900 proceeds to box 950, where the icon is tilted to the right, as described above for the "tilt left" response. This may include rotation, translation, or any combination of rotation and translation of the example icon, and then bounces back to its original position at the top portion of 3D space. Simultaneously, the right tilt gesture mapped to the functionality performed by the display device is executed, and method 900 proceeds to query box 955. However, if the response at query box 945 is "No", then method 900 proceeds directly to query box 955, where it is determined whether a squeeze gesture by the user to exit the air display mode has been detected. If the response at query box 955 is "Yes", then method 900 proceeds to box 960, where the icon disappears, and method 900 terminates. Alternatively, as described above, method 900 may not terminate at box 960, but instead return to box 910 and wait for the next grab gesture.

[0096] However, if "No" is returned at prompt 955, and the user therefore wishes to continue operating the display device in air display mode, then method 900 returns to prompt 935 to determine whether a left tilt gesture has been detected. Method 900 can therefore cycle through prompts 93, 940, 945, 950, and 955, terminating air display mode as soon as "Yes" is returned at prompt 955, provided the user wishes to operate the display device in air display mode.

[0097] Regarding which functions can be mapped to the "tilt left" and "tilt right" contactless gestures, it's important to note that simplicity of operation is generally desired in air display modes. Therefore, in most embodiments, for example, air interaction with the assistant icon will not be used for composing emails. Thus, in one or more embodiments, continuous interaction with the icons described in method 900 can be equivalent to up / down or left / right soft keys to allow the user to navigate through various predefined UI states. For example, at the highest level, actions could include flipping from one application icon to another. At a lower level, such as in an email application, actions could include flipping from one email to another. Or, for example, in a music player application, actions could include changing from one song to another, or even increasing or decreasing the volume.

[0098] Combination Figure 9 The example method 900, the example ordering of various boxes, and which boxes are included are understood to be exemplary only. In other example embodiments of the disclosed method, one can refer to, for example... Figure 9 The order shown changes the order of any boxes, allowing boxes to be executed in parallel, and excludes some boxes. Therefore, for example, using a gesture to initially enter the air display mode preview is optional and can be omitted. Thus, a single gesture (such as, for example, a grab or pull, or other gesture) can be interpreted as a command to directly enter the air display mode without requiring a prior preview mode. Additionally, for example, tilting left, tilting right, and air display termination gestures can be detected in different orders or in parallel, making the method unnecessary to perform actions such as re-checking the tilting left gesture (e.g., as shown in query box 935). Figure 9 The loop shown is through boxes 935, 945, and 955.

[0099] In one or more embodiments, a given display device including the exemplary system may be handheld, installed in a vehicle, installed in a public kiosk, used in a private kitchen, etc. Furthermore, the application used in conjunction with contactless gestures may be an AR application, or may not, as long as it involves the aerial display projection of images generated by the application, such as, for example, basic computer controls, as described above. Figure 2C and 2D As shown in the image.

[0100] In one or more embodiments, the non-contact gestures may differ from the examples described above. For example, instead of a grasping gesture, a "scoop" gesture could be used, which would also appear very intuitive to the user. In such embodiments, the scoop response needs to be precisely defined to, for example, eliminate ambiguity from other more common gestures such as swiping. For example, the conditions for registering such alternative gestures can vary and additional time / space lags can be employed. Thus, for example, when switching from a normal 2D display mode to a 3D air display mode, as described above, the switching criterion may not be a clearly defined "one-point" threshold. For example, taking a squeeze gesture as an example, it first needs to pass a "W1" test before even applying a "W2". In other words, if the user's hand is too far from the display screen to have a good baseline W1 for comparison with a larger W2, then it is not even necessary to test the squeeze that has been performed. As another example, see reference Figure 5 As described, grasping and non-grabbing gestures have different distance thresholds “D1” and “D2”, making the two gestures non-interchangeable.

[0101] Note that while any gesture can be used to initiate or terminate a 3D air display mode, it may be advantageous to use gestures that meet certain criteria to prevent unintentional triggering of mode changes. For example, gestures that are simple enough to be performed and easy to remember by the user, hand movements that require a minimum signal strength to be detected, a set of gestures that each have a capacitive response, or gestures with other suitable criteria can be used.

[0102] In alternative embodiments, a second camera, one or more IR sensors, ultrasonic sensors, and / or other types of sensors, incorporating capacitive sensing, can be used to detect or initiate the detection of the disclosed contactless gesture. However, it should be noted that this may require increased processing power and hardware size, along with associated costs.

[0103] The embodiments and examples set forth herein are presented in order to best explain embodiments according to the present technology and its particular applications, thereby enabling those skilled in the art to make and use this disclosure. However, those skilled in the art will recognize that the foregoing descriptions and examples have been presented for illustrative and exemplary purposes only. As the descriptions set forth are not intended to be exhaustive or to limit this disclosure to its precise forms.

[0104] In view of the foregoing, the scope of this disclosure is defined by the appended claims.

Claims

1. A method for interactively displaying images in a 3D space near a display device, comprising: Display graphics and / or text at a predefined original location in the 3D space near the display device; By analyzing the first capacitive response on the display device, it is determined that the user has performed a first interactive non-contact gesture on the graphic and / or text at the predefined original location; In response to determining that the user has performed the first interactive non-contact gesture on the graphic and / or text, the movement of the graphic and / or text to a second position in the 3D space is displayed, and then after a predefined time, the graphic and / or text is returned to the predefined original position, wherein the predefined original position is different from the second position; By analyzing the second capacitive response on the display device, it is determined that the user has performed a second interactive non-contact gesture on the graphic and / or text at the predefined original location; as well as In response to determining that the user has performed the second interactive non-contact gesture, the graphic and / or text is displayed to move to a third position in the 3D space, and then after the predefined time, the graphic and / or text is returned to the predefined original position, wherein the predefined original position is different from the third position.

2. The method of claim 1, wherein each of the first interactive contactless gesture and the second interactive contactless gesture is mapped to a corresponding command of an application running on the display device.

3. The method of claim 1, wherein the first interactive non-contact gesture is a rightward tilt gesture, and in response to the first interactive non-contact gesture, the graphic and / or text is displayed to tilt to the right and, after the predefined time, the graphic and / or text returns to the predefined original position; and The second interactive non-contact gesture is a left tilt gesture, and in response to the second interactive non-contact gesture, the graphic and / or text is displayed to tilt to the left and, after the predefined time, the graphic and / or text returns to the predefined original position.

4. The method according to claim 1, further comprising: By analyzing the third capacitive response on the display device, it is determined that the user has performed a termination non-contact gesture in the 3D space; as well as In response to determining that the user has performed the termination non-contact gesture in the 3D space, the display of the graphics and / or text is stopped.

Images