Method and electronic device for enabling virtual input on electronic device

Abstract

ABSTRACT “Method and electronic device for enabling virtual input on electronic device” Embodiments herein disclose a method for enabling virtual input by an electronic device (100). The method includes measuring, by the electronic device (100), sensor data corresponding to a gesture performed by a user on a surface area (400). Further, the method includes detecting, by the device (100), a virtual input event by a finger (300) of the user based on the sensor data. Further, the method includes determining, by the electronic device (100), a portion at which the gesture is performed by the user on the surface area (400) based on the virtual input event. Further, the method includes detecting, by the electronic device (100), a key (200) corresponding to the portion at which the gesture is performed. Further, the method includes performing, by the electronic device (100), an action based on the detected key (200). The virtual input event corresponds to a typing event or an application control event. FIG. 1

Description

FIELD OF INVENTION

[0001] The present disclosure relates to a virtual input on anelectronic device.BACKGROUND OF INVENTION

[0002] In general, various mechanisms are used for typing an inputon a keyboard. The user of an electronic device faces many challengeswhile using a virtual and hard keyboard. The challenges can be, forexample, but not limited to a slow typing, user intention while typing, smallkeys, non-custom keyboard and an external hardware requirement. Manyconventional methods and electronic devices have been proposed for typingan input on a keyboard, but these conventional methods and electronicdevice may have disadvantages in terms of robustness, reliability, operationdependency, time, cost, complexity, design, hardware components usage,size and so on.

[0003] Thus, it is desired to address the above mentioneddisadvantages or other shortcomings or at least provide a useful alternative.OBJECT OF INVENTION

[0004] The principal object of the embodiments herein is to providea method and an electronic device for enabling virtual input so as toprovide an enhanced user experience for typing, so that a user of theelectronic device can type smartly and quickly without seeing on keys at acost effective manner and without using additional hardware.

[0005] Another object of the embodiments herein is to enable thevirtual typing by finger tapping / swiping / handwriting on any free surface.The electronic device will detect finger movements and notify the detectedfinger movements on a virtual input controller and the virtual inputcontroller will map finger tips to the keyboard map for hassle free typing.This results in enhancing a keyboard typing experience as per userrequirements.

[0006] Another object of the embodiments herein is to provide thevirtual input on one or more application executing on the electronic deviceto control various functions (e.g., scrolling the application, a pausing avideo in the electronic device or the like) on the one or more application byusing finger tapping / swiping / handwriting on any free surface. Theelectronic device will detect finger movements and notify the detectedfinger movements on the virtual input controller and the virtual inputcontroller will map finger tips to the keyboard map for hassle free virtualinput on the one or more application. This results in enhancing theapplication control as per user requirements without using any externalhardware in a cost effective manner.SUMMARY

[0007] Accordingly, embodiments herein disclose a method forenabling virtual input by an electronic device. The method includesmeasuring, by the electronic device, sensor data corresponding to a gestureperformed by a user on a surface area. Further, the method includesdetecting, by the electronic device, a virtual input event by a finger of theuser based on the sensor data. Further, the method includes determining, bythe electronic device, a portion at which the gesture is performed by theuser on the surface area based on the virtual input event. Further, themethod includes detecting, by the electronic device, a key corresponding tothe portion at which the gesture is performed. Further, the method includesperforming, by the electronic device, an action based on the detected key.

[0008] In an embodiment, the surface area is one of a normalsurface area and an artificial surface area.

[0009] In an embodiment, the normal surface area is one of a glasssurface, a rubber surface, a plastic surface, a wooden surface, and amattress surface.

[0010] In an embodiment, the artificial surface area is one of aprinted keyboard and a handwritten keyboard on the normal surface area.

[0011] In an embodiment, the sensor data includes an accelerometersensor data, a prism data and an ultrasonic data.

[0012] In an embodiment, the sensor data is obtained from at leastone of a camera, a prism, and Human body communication (HBC).

[0013] In an embodiment, the typing event includes a tap of thefinger, a swipe of the finger, and a handwriting of the user.

[0014] In an embodiment, measuring, by the electronic device, thesensor data corresponding to the gesture performed by the user on thesurface area includes automatically activating, by the electronic device, acamera of the electronic device, detecting, by the electronic device, whetherthe surface area corresponds to a normal surface area or an artificial surfacearea displayed in a field of view of the camera, and performing, by theelectronic device, one of: capturing an image of the normal surface areadisplayed within the field of view of the camera in response to determiningthat the surface area corresponds to the normal surface, generating a virtualarea by mapping a keyboard layout of the electronic device to the image ofthe normal surface area, detecting the gesture performed by the user in thevirtual area, and measuring the sensor data when the gesture is performedin the virtual area, and capturing an image of the artificial surface areadisplayed within the field of view of the camera in response to determiningthat the surface area corresponds to the artificial surface, generating akeyboard layout provided in the artificial surface by processing the imageof the artificial surface area, and measuring the sensor data when thegesture is performed on the normal surface area.

[0015] In an embodiment, mapping the keyboard layout of theelectronic device to the image of the normal surface area includesdetecting, by the electronic device, one of a left hand of the user, a righthand of the user and both left and right hands of the user, and performing,by the electronic device, one of mapping a left hand keyboard layout to theimage of the normal surface area in response to detecting the left hand ofthe user, mapping a right hand keyboard layout to the image of the normalsurface area in response to detecting the right hand of the user, andmapping a split hand keyboard layout to the image of the normal surfacearea in response to detecting both the left hand and the right hand of theuser.

[0016] In an embodiment, detecting, by the electronic device, thetyping event by the finger of the user based on the sensor data includesapplying, by the electronic device, a machine learning model to detect thefinger of the user based on the gesture performed by the user on the surfacearea, applying, by the electronic device, the machine learning model todetect a tip of the finger of the user based on the gesture performed by theuser on the surface area, and detecting, by the electronic device, the typingevent performed by the tip of the finger of the user based on the sensordata.

[0017] In an embodiment, detecting, by the electronic device, thekey corresponding to the portion includes detecting, by the electronicdevice, whether the surface area corresponds to a normal surface area or anartificial keyboard, and performing, by the electronic device, one of:mapping the portion at which the gesture is performed with the key of akeyboard of the electronic device in response to determining that thesurface area corresponds to the normal surface area, and detecting the keycorresponding to the portion based on the mapping, and mapping theportion at which the gesture is performed to the key of a key map providedin the artificial surface area in response to determining that the surface areacorresponds to the artificial surface area, and detecting the keycorresponding to the portion based on the mapping.

[0018] Accordingly embodiments herein disclose an electronicdevice for enabling virtual input. The electronic device includes a sensorcoupled to a memory and a processor. The sensor is configured to measuresensor data corresponding to a gesture performed by a user on a surfacearea. A virtual input controller is coupled to the memory and the processor.The virtual input controller is configured to detect a virtual input event by afinger of the user based on the sensor data. Further, the virtual inputcontroller is configured to determine a portion at which the gesture isperformed by the user on the surface area based on the virtual input event.Further, the virtual input controller is configured to detect a keycorresponding to the portion at which the gesture is performed. Further, thevirtual input controller is configured to perform an action based on thedetected key.

[0019] These and other aspects of the embodiments herein will bebetter appreciated and understood when considered in conjunction with thefollowing description and the accompanying drawings. It should beunderstood, however, that the following descriptions, while indicatingpreferred embodiments and numerous specific details thereof, are given byway of illustration and not of limitation. Many changes and modificationsmay be made within the scope of the embodiments herein without departingfrom the spirit thereof, and the embodiments herein include all suchmodifications.BRIEF DESCRIPTION OF FIGURES

[0020] This method is illustrated in the accompanying drawings,throughout which like reference letters indicate corresponding parts in thevarious figures. The embodiments herein will be better understood from thefollowing description with reference to the drawings, in which:

[0021] FIG. 1 is an example scenario in which an electronic deviceenables a virtual input, according to an embodiment as disclosed herein;

[0022] FIG. 2 shows various hardware components of the electronicdevice, according to an embodiment as disclosed herein;

[0023] FIG. 3 is a flow chart illustrating a method for enabling thevirtual input on the electronic device, according to an embodiment asdisclosed herein;

[0024] FIG. 4 is an example scenario in which the electronic devicedisplays corresponding letters on a display by detecting a location of afinger position on a virtual keyboard and mapping to a keyboard key-mapto realize corresponding key presses on the virtual keyboard, according toan embodiment as disclosed herein;

[0025] FIG. 5 is an example scenario in which a raw imagetransformation data, a contour detection and OCR data and a fingertracking data are depicted, according to an embodiment as disclosed herein;

[0026] FIG. 6 is another example scenario in which the electronicdevice displays corresponding letters on a display by detecting a location ofa finger position on a paper keyboard and mapping to a keyboard key-mapto realize the corresponding key press on the paper keyboard, according toan embodiment as disclosed herein;

[0027] FIG. 7 is another example scenario in which the electronicdevice displays corresponding letters on the display based on a swipegesture on the keyboard of the electronic device, according to anembodiment as disclosed herein;

[0028] FIG. 8 is another example scenario in which the electronicdevice displays corresponding letters on the display by typing using aprinted keyboard on a surface, according to an embodiment as disclosedherein;

[0029] FIG. 9 is another example scenario in which the electronicdevice displays corresponding letters on the display by typing using fingertracking, according to an embodiment as disclosed herein;

[0030] FIG. 10 is another example scenario in which the electronicdevice displays corresponding letters on the display by swiping on theprinted keyboard to type, according to an embodiment as disclosed herein;

[0031] FIG. 11 is another example scenario in which the electronicdevice displays corresponding letters on the display by swiping on anysurface to type, according to an embodiment as disclosed herein;

[0032] FIG. 12 is another example scenario in which the electronicdevice displays corresponding letters on the display by recognitionhandwriting using a pen, according to an embodiment as disclosed herein;

[0033] FIG. 13 is another example scenario in which the electronicdevice displays corresponding letters on the display by typing using fingertracking of two hands, according to an embodiment as disclosed herein;

[0034] FIG. 14 is another example scenario in which the electronicdevice displays corresponding letters on the display by typing on a customkeyboard, according to an embodiment as disclosed herein;

[0035] FIG. 15 is an example scenario in which the electronicdevice maps a left hand keyboard layout to an image of a normal surfacearea, according to an embodiment as disclosed herein;

[0036] FIG. 16 is an example scenario in which the electronicdevice maps a right hand keyboard layout to an image of the normal surfacearea, according to an embodiment as disclosed herein;

[0037] FIG. 17 is an example scenario in which the electronicdevice maps a normal keyboard layout to the image of the normal surfacearea, according to an embodiment as disclosed herein;

[0038] FIG. 18 is another example scenario in which the electronicdevice maps the split hand keyboard layout to the image of the normalsurface area, according to an embodiment as disclosed herein;

[0039] FIG. 19 is an example scenario in which the electronicdevice identifies the finger events on the virtual keyboard using HumanBody Communication (HBC), according to an embodiment as disclosedherein;

[0040] FIG. 20 is another example scenario in which the electronicdevice displays corresponding letters on the display by HBC, according toan embodiment as disclosed herein;

[0041] FIG. 21 is an example scenario in which the electronicdevice identifies the finger events on the virtual keyboard using a prism,according to an embodiment as disclosed herein;

[0042] FIG. 22 is another example scenario in which the electronicdevice displays corresponding letters on the display using a predictiontechnique, according to an embodiment as disclosed herein;

[0043] FIG. 23 is an example scenario in which the electronicdevice controls a gallery and a video application in a multi windowscenario by detecting a key corresponding to the portion at which the tap isperformed on a surface area, according to an embodiment as disclosedherein;

[0044] FIG. 24 is an example scenario in which the electronicdevice controls a scrolling operation on a music application by detectingthe key corresponding to the portion at which the tap is performed on thesurface area, according to an embodiment as disclosed herein;

[0045] FIG. 25 is an example scenario in which the electronicdevice controls a touch pad operation on a music application by detecting akey corresponding to the portion at which the tap is performed on thesurface area, according to an embodiment as disclosed herein; and

[0046] FIG. 26 is an example scenario in which the electronicdevice edits links on a webpage by using a virtual input, according to anembodiment as disclosed herein.DETAILED DESCRIPTION OF INVENTION

[0047] The embodiments herein and the various features andadvantageous details thereof are explained more fully with reference to thenon-limiting embodiments that are illustrated in the accompanyingdrawings and detailed in the following description. Descriptions of wellknowncomponents and processing techniques are omitted so as to notunnecessarily obscure the embodiments herein. Also, the variousembodiments described herein are not necessarily mutually exclusive, assome embodiments can be combined with one or more other embodimentsto form new embodiments. The term "or" as used herein,refers to a nonexclusive or, unless otherwise indicated. The examples used herein areintended merely to facilitate an understanding of ways in which theembodiments herein can be practiced and to further enable those skilled inthe art to practice the embodiments herein. Accordingly, the examplesshould not be construed as limiting the scope of the embodiments herein.

[0048] As is traditional in the field, embodiments may be describedand illustrated in terms of blocks which carry out a described function orfunctions. These blocks, which may be referred to herein as units ormodules or the like, are physically implemented by analog or digitalcircuits such as logic gates, integrated circuits, microprocessors,microcontrollers, memory circuits, passive electronic components, activeelectronic components, optical components, hardwired circuits, or the like,and may optionally be driven by firmware and software. The circuits may,for example, be embodied in one or more semiconductor chips, or onsubstrate supports such as printed circuit boards and the like. The circuitsconstituting a block may be implemented by dedicated hardware, or by aprocessor (e.g., one or more programmed microprocessors and associatedcircuitry), or by a combination of dedicated hardware to perform somefunctions of the block and a processor to perform other functions of theblock. Each block of the embodiments may be physically separated intotwo or more interacting and discrete blocks without departing from thescope of the invention. Likewise, the blocks of the embodiments may bephysically combined into more complex blocks without departing from thescope of the invention

[0049] The accompanying drawings are used to help easilyunderstand various technical features and it should be understood that theembodiments presented herein are not limited by the accompanyingdrawings. As such, the present disclosure should be construed to extend toany alterations, equivalents and substitutes in addition to those which areparticularly set out in the accompanying drawings. Although the terms first,second, etc. may be used herein to describe various elements, theseelements should not be limited by these terms. These terms are generallyonly used to distinguish one element from another.

[0050] Accordingly, embodiments herein achieve a method forenabling virtual input by an electronic device. The method includesmeasuring, by the electronic device, sensor data corresponding to a gestureperformed by a user on a surface area. Further, the method includesdetecting, by the electronic device, the virtual input by a finger of the userbased on the sensor data. Further, the method includes determining, by theelectronic device, a portion at which the gesture is performed by the user onthe surface area based on the virtual input. Further, the method includesdetecting, by the electronic device, a key corresponding to the portion atwhich the gesture is performed. Further, the method includes performing,by the electronic device, an action based on the detected key. The virtualinput corresponds to a virtual typing event and virtually performing anaction on an application (e.g., scrolling the application, a pausing a video inthe electronic device or the like).

[0051] Unlike conventional methods and systems, the method canbe used to enable the virtual typing so as to provide an enhanced userexperience for typing, so that a user of the electronic device can typesmartly and quickly without seeing on keys at a cost effective manner andwithout using additional hardware. The method can be used to virtuallyperform the action on the application (e.g., scrolling the application, apausing a volume of the application or the like.) in an easy manner withoutusing additional hardware.

[0052] The proposed method can be used to enable the virtualtyping by finger tapping / swiping on any free surface. The electronic devicewill detect the finger movements and notify the detected finger movementson a virtual input controller and the virtual input controller will map fingertips to the keyboard map for hassle free typing. This results in enhancing akeyboard typing experience as per user requirements. The proposed methodcan be used to enable the virtual typing by finger tapping / swiping on anyfree surface at a cost effective manner and without additional hardware.

[0053] Referring now to the drawings, and more particularly toFIGS. 1 through 26, there are shown preferred embodiments.

[0054] FIG. 1 is an example scenario in which an electronic device(100) enables a virtual input, according to an embodiment as disclosedherein. The electronic device (100) can be, for example, but not limited to acellular phone, a smart phone, a Personal Digital Assistant (PDA), a tabletcomputer, a laptop computer, an Internet of Things (IoT), a virtual realitydevice, a smart phone, a flexible electronic device, a curved electronicdevice, and a foldable electronic device.

[0055] Consider a scenario, a user of the electronic device (100)wants to type on the electronic device (100) using a virtual keyboard. Basedon a proposed method, the electronic device (100) automatically activates acamera (102). After activating the camera (102), the electronic device (100)detects whether a surface area (400) corresponds to a normal surface areaor an artificial surface area displayed in a field of view of the camera (102).The normal surface area can be, for example, but not limited to a glasssurface, a rubber surface, a plastic surface, a metal surface, a woodensurface, and a mattress surface. The artificial surface area can be, forexample, but not limited to a printed keyboard and a handwritten keyboardon the normal surface area. The proposed method is applicable to all thesurfaces which can detect vibrations.

[0056] If the surface area (400) corresponds to the artificial surfacearea displayed in the field of view of the camera (102) then, the camera(102) captures an image of the artificial surface area displayed within afield of view of the camera (102). Further, the electronic device (100) mapsa keyboard layout provided in the artificial surface by processing the imageof the artificial surface area. Further, the electronic device (100) measures sensor data when a gesture is performed on the artificial surface area. Thesensor data can be, for example but not limited to, an accelerometer sensordata, a gyroscope sensor data and an ultrasonic data. Further, the sensordata is obtained from at least one of a camera, a prism, and Human bodycommunication (HBC).

[0057] If the surface area (400) corresponds to the normal surfacearea displayed in the field of view of the camera (102) then, the camera(102) captures the image of the normal surface area displayed within thefield of view of the camera (102). Further, the electronic device (100)detects a left hand of the user, a right hand of the user or both left and righthands of the user. If the left hand of the user is detected on the normalsurface area displayed within the field of view of the camera (102) then, theelectronic device (100) maps the left hand keyboard layout to the image ofthe normal surface area. The operations and functions of the left handkeyboard layout is explained in the FIG. 15.

[0058] If the right hand of the user is detected on the normal surfacearea displayed within the field of view of the camera (102) then, theelectronic device (100) maps the right hand keyboard layout to the image ofthe normal surface area. The operations and functions of the right handkeyboard layout is explained in the FIG. 16. If the left hand and right handof the user is detected on the normal surface area displayed within the fieldof view of the camera (102) then, the electronic device (100) maps the splithand keyboard layout or the normal keyboard layout to the image of thenormal surface area. The operations and functions of the split handkeyboard layout is explained in the FIG. 18. The split hand keyboard layoutor the normal keyboard layout are determined by identifying a distancebetween one hand to another hand of the user. If the distance between theone hand to another hand of the user exceeds a predefine threshold then, theelectronic device (100) activates the split hand keyboard layout. If thedistance between the one hand to another hand of the user does not exceedthe predefine threshold then, the electronic device (100) activates thenormal keyboard layout. The predefine threshold is set by the user of theelectronic device (100) or an original Equipment manufacturer (OEM).

[0059] Further, the electronic device (100) generates the virtualarea. After generating the virtual area, the electronic device (100) detectsthe gesture performed by the user in the virtual area. Further, the electronicdevice (100) measures the sensor data when the gesture is performed in thevirtual area.

[0060] Further, the electronic device (100) applies the machinelearning model to detect one or more finger (300) of the user based on thegesture performed by the user on the surface area (400). Further, theelectronic device (100) applies the machine learning model to detect a tipof the finger (300) of the user based on the gesture or the event performedby the user on the surface area (400). The gesture or the event can be, forexample, but not limited to a swipe gesture and a hover gesture. Further,the electronic device (100) detects a typing event performed by the tip ofthe finger (300) of the user based on the sensor data. The typing event canbe, for example, but not limited to a tap of the finger (300), a swipe of the5 finger (300), and a handwriting of the user. Further, the electronic device(100) determines the portion at which the gesture is performed by the useron the surface area (400).

[0061] After determining the portion at which the gesture isperformed by the user on the surface area (400), the electronic device (100)detects whether the surface area corresponds to the normal surface area orthe artificial surface area. If the surface area corresponds to the normalsurface area then, the electronic device (100) maps the portion at which thegesture is performed with the key (200) of a keyboard of the electronicdevice (100). Further, the electronic device (100) detects the key (200) corresponding to the portion based on the mapping.

[0062] If the surface area corresponds to the artificial surface areathen, the electronic device (100) maps the portion at which the gesture isperformed to the key (200) of a key map provided in the artificial surfacearea. Further, the electronic device (100) detects the key corresponding tothe portion based on the mapping. Further, the electronic device (100)performs the action based on the detected key (200).

[0063] FIG. 2 shows various hardware components of the electronicdevice (100), according to an embodiment as disclosed herein. Theelectronic device (100) includes the camera (102), a communicator (104), amemory (106), a processor (108), a display (110), a virtual input controller(112), a sensor (114), a machine learning controller (116) and a prismcontroller (118). The sensor (114) can be, for example, but not limited to anaccelerometer sensor (114a) and an ultrasonic sensor (114b). The processor(108) is coupled with the camera (102), the communicator (104), thememory (106), the display (110), the virtual input controller (112), thesensor (114), the machine learning controller (116) and the prism controller(118). The prism controller (118) is used to obtain the sensor data.

[0064] The virtual input controller (112) is configured toautomatically activate the camera (102). After activating the camera (102),the virtual input controller (112) is configured to detect whether the surfacearea (400) corresponds to the normal surface area or the artificial surfacearea displayed in the field of view of the camera (102). If the surface area(400) corresponds to the artificial surface area displayed in the field of viewof the camera (102) then, the camera (102) captures the image of theartificial surface area displayed within the field of view of the camera(102). Further, the virtual input controller (112) is configured to map thekeyboard layout provided in the artificial surface by processing the imageof the artificial surface area. Further, the virtual input controller (112) isconfigured to measure the sensor data, using the sensor (114), when thegesture is performed on the artificial surface area.

[0065] If the surface area (400) corresponds to the normal surfacearea displayed in the field of view of the camera (102) then, the camera(102) captures the image of the normal surface area displayed within thefield of view of the camera (102). Further, the virtual input controller (112)is configured to detect the left hand of the user, the right hand of the user orboth left and right hands of the user. If the left hand of the user is detectedon the normal surface area displayed within the field of view of the camera(102) then, the virtual input controller (112) is configured to map the lefthand keyboard layout to the image of the normal surface area. If the righthand of the user is detected on the normal surface area displayed within thefield of view of the camera (102) then, the virtual input controller (112) isconfigured to map the right hand keyboard layout to the image of thenormal surface area. If the left hand and right hand of the user is detectedon the normal surface area displayed within the field of view of the camera(102) then, virtual input controller (112) is configured to map the split handkeyboard layout to the image of the normal surface area.

[0066] Further, the virtual input controller (112) is configured togenerate the virtual area and detect the gesture performed by the user in thevirtual area. Further, the virtual input controller (112) is configured tomeasure the sensor data when the gesture is performed in the virtual area.Further, the virtual input controller (112) is configured to apply themachine learning model to detect the finger (300) of the user based on thegesture performed by the user on the surface area (400) using the machinelearning controller (116). The machine learning model identifies the gesturethat is being performed in a touch with the surface in close proximity to theelectronic device (100). Further, the machine learning controller (116) isconfigured to apply the machine learning model to detect a tip of the finger(300) of the user when the tip of the finger is not detected based on thegesture performed by the user on the surface area (400). Further, the virtualinput controller (112) is configured to detect the typing event performed bythe tip of the finger (300) of the user based on the sensor data. Further, thevirtual input controller (112) is configured to determine the portion atwhich the gesture is performed by the user on the surface area (400).

[0067] Further, the virtual input controller (112) is configured todetect whether the surface area corresponds to the normal surface area orthe artificial surface area. If the surface area corresponds to the normalsurface area then, the virtual input controller (112) is configured to map theportion at which the gesture is performed with the key (200) of thekeyboard of the electronic device (100). Further, the virtual input controller(112) is configured to detect the key (200) corresponding to the portionbased on the mapping.

[0068] If the surface area corresponds to the artificial surface areathen, the virtual input controller (112) is configured to map the portion atwhich the gesture is performed to the key (200) of the key map provided inthe artificial surface area. Further, the virtual input controller (112) isconfigured to detect the key corresponding to the portion based on themapping. Further, the virtual input controller (112) is configured to performthe action based on the detected key (200).

[0069] The virtual input controller (112) also includes a handdetector model (not shown), a hand landmark model (not shown), and agesture recognizer model (not shown). The hand detector model operateson full image and returns oriented hand bounding box. The hand landmarkmodel operates on cropped image region defined by a hand detector andreturns high fidelity 3D hand key points (i.e., finger tips location & distancebetween the fingers). The gesture recognizer model classifies thepreviously computed key points into a discrete set of gestures.

[0070] Further, the accelerometer sensor (114a) is used to measureacceleration forces. The forces may be static, like the continuous force ofgravity or, as is the case with many electronic devices (100), dynamic tosense movement or vibrations. The sensor (114) provides the threecoordinate axis x, y and z of the vibration. The finger tap detection on thesurface is detected using accelerometer vibrations. The machine learningmodel is used to calibrate and prune for false tap vibrations using themachine learning controller (116).

[0071] Further, the ML model includes a calibration model, a tappruning model, and a tap recognizer model. The calibration model thatoperates on the all vibrations and calibrates the electronic device (100) forfalse vibrations based on the environment. The tap pruning model operateson the finger taps defined by the calibration model. The tap pruning modellearns the user key vibration mapping. In an example, the user key pressintensity is different for each key. The tap pruning model will learn the userkey press intensity to identify the desired key press. The tap recognizermodel classifies the previously computed taps configuration into a discreteset of keys.

[0072] The electronic device (100) also identifies the finger eventson the virtual keyboard using HBC and prism. The HBC related operationis explained in the FIG. 19. The prism related operation is explained in theFIG. 21.

[0073] Further, the electronic device (100) also identifies the fingerevents on the virtual keyboard using an ultrasonic event transfer method.The ultrasonic event transfer method transmits ultrasonic waves andreceives echoes reflected from the palms and fingers. In the ultrasonicevent transfer method, a pulsed Doppler radar signal processing techniqueis used to obtain time-sequential range-Doppler features from ultrasonicwaves, measure objects' precise distances and velocities simultaneouslythrough a single channel.

[0074] Further, the electronic device (100) also identifies the fingerevents on the virtual keyboard using a State-transition based hiddenMarkov model (HMM) method for micro dynamic gesture recognition.

[0075] The processor (108) is configured to execute instructionsstored in the memory (106) and to perform various processes. Thecommunicator (104) is configured for communicating internally betweeninternal hardware components and with external devices via one or morenetworks.

[0076] The memory (106) also stores instructions to be executed bythe processor (108). The memory (106) may include non-volatile storageelements. Examples of such non-volatile storage elements may includemagnetic hard discs, optical discs, floppy discs, flash memories, or forms ofelectrically programmable memories (EPROM) or electrically erasable andprogrammable (EEPROM) memories. In addition, the memory (106) may,in some examples, be considered a non-transitory storage medium. Theterm "non-transitory" may indicate that the storage medium is notembodied in a carrier wave or a propagated signal. However, the term"non-transitory" should not be interpreted that the memory (106) is nonmovable.In certain examples, a non-transitory storage medium may storedata that can, over time, change (e.g., in Random Access Memory (RAM)or cache).

[0077] Further, at least one of the plurality of hardware componentsmay be implemented through an artificial intelligence (AI) model. Afunction associated with AI may be performed through the non-volatilememory, the volatile memory, and the processor (108). The processor (108)may include one or a plurality of processors. At this time, one or a pluralityof processors may be a general purpose processor, such as a centralprocessing unit (CPU), an application processor (AP), or the like, agraphics-only processing unit such as a graphics processing unit (GPU), avisual processing unit (VPU), and / or an AI-dedicated processor such as aneural processing unit (NPU).

[0078] The one or a plurality of processors control the processing ofthe input data in accordance with a predefined operating rule or artificialintelligence (AI) model stored in the non-volatile memory and the volatilememory. The predefined operating rule or artificial intelligence model isprovided through training or learning.

[0079] Here, being provided through learning means that, byapplying a learning algorithm to a plurality of learning data, a predefinedoperating rule or AI model of a desired characteristic is made. The learningmay be performed in a device itself in which AI according to anembodiment is performed, and / or may be implemented through a separateserver / system.

[0080] The AI model may consist of a plurality of neural networklayers. Each layer has a plurality of weight values, and performs a layeroperation through calculation of a previous layer and an operation on aplurality of weights. Examples of neural networks include, but are notlimited to, convolutional neural network (CNN), deep neural network(DNN), recurrent neural network (RNN), restricted Boltzmann Machine(RBM), deep belief network (DBN), bidirectional recurrent deep neuralnetwork (BRDNN), generative adversarial networks (GAN), and deep Qnetworks.

[0081] The learning algorithm is a method for training apredetermined target device (for example, a robot) using a plurality oflearning data to cause, allow, or control the target device to make adetermination or prediction. Examples of learning algorithms include, butare not limited to, supervised learning, unsupervised learning, semisupervised learning, or reinforcement learning.

[0082] Although the FIG. 2 shows various hardware components ofthe electronic device (100) but it is to be understood that otherembodiments are not limited thereon. In other embodiments, the electronicdevice (100) may include less or more number of components. Further, thelabels or names of the components are used only for illustrative purposeand does not limit the scope of the invention. One or more components canbe combined together to perform same or substantially similar function toenable the virtual input on the electronic device (100).

[0083] FIG. 3 is a flow chart (S300) illustrating a method forenabling virtual input on the electronic device (100), according to anembodiment as disclosed herein. The operations (S302-S340) areperformed by the virtual input controller (112).

[0084] At S302, the method includes automatically activating thecamera (102) of the electronic device (100). At S304, the method includesdetecting whether the surface area corresponds to the normal surface areaor the artificial surface area displayed in a field of view of the camera(102). If the surface area (400) corresponds to the artificial surface areadisplayed in the field of view of the camera (102) then, at S306, the methodincludes capturing the image of the artificial surface area displayed withinthe field of view of the camera (102). At S308, the method includesmapping the keyboard layout provided in the artificial surface byprocessing the image of the artificial surface area.

[0085] At S310, the method includes capturing the image of the normal surface area displayed within the field of view of the camera (102).At S312, the method includes mapping the keyboard layout provided in thenormal surface. At S314, the method includes detecting one of a left handof the user, a right hand of the user and both left and right hands of the user.If the left hand of the user is detected on the normal surface area displayedwithin the field of view of the camera (102) then, at S316, the methodincludes mapping the left hand keyboard layout to the image of the normalsurface area. If the right hand of the user is detected on the normal surfacearea displayed within the field of view of the camera (102) then, at S318,the method includes mapping the right hand keyboard layout to the imageof the normal surface area. If the left hand and right hand of the user isdetected on the normal surface area displayed within the field of view ofthe camera (102) then, at S320, the method includes mapping the split handkeyboard layout to the image of the normal surface area.

[0086] At S322, the method includes generating the virtual area. AtS324, the method includes detecting the gesture performed by the user inthe virtual area. At S326, the method includes measuring the sensor datawhen the gesture is performed in the virtual area. At S328, the methodincludes applying the machine learning model to detect the finger of theuser based on the gesture performed by the user on the surface area (400).At S330, the method includes applying the machine learning model todetect a tip of the finger (300) of the user based on the gesture performedby the user on the surface area (400). At S332, the method includesdetecting the typing event performed by the tip of the finger (300) of theuser based on the sensor data. At S334, the method includes determiningthe portion at which the gesture is performed by the user on the surface area(400).

[0087] At S336, the method includes mapping the portion at whichthe gesture is performed with the key (200) of the keyboard of theelectronic device (100). At S338, the method includes detecting the key(200) corresponding to the portion based on the mapping. At S340, themethod includes performing the action based on the detected key (200).The action can be a virtual typing, the application control or the like.

[0088] In the proposed method, the user of the electronic device(100) can make custom layout and calibrate his / her own keyboard in acustom typing mode. The custom typing mode can include custom keys(e.g., Predictions, play, pause etc.), multilingual keyboard (English, Hindikeys, Korean keys, etc.).

[0089] The method can be extendable to any type of typing overany tangible surface and just slide your fingers around the any tangiblesurface.

[0090] The various actions, acts, blocks, steps, or the like in theflow chart (S300) may be performed in the order presented, in a differentorder or simultaneously. Further, in some embodiments, some of theactions, acts, blocks, steps, or the like may be omitted, added, modified,skipped, or the like without departing from the scope of the invention.

[0091] FIG. 4 is an example scenario in which the electronic device(100) displays corresponding letters on a display (110) by detecting alocation of a finger position on the virtual keyboard and mapping to thekeyboard key-map to realize the corresponding key press on the virtualkeyboard, according to an embodiment as disclosed herein.

[0092] Consider, the user of the electronic device (100) draws thekeys specific to his / her need or a keyboard layout of his / her choice, theelectronic device (100) automatically identifies layout (i.e., creating virtualkeyboard). Based on the proposed method, a front camera of the electronicdevice identifies a keyboard design drawn by the user and understands alayout of the keyboard design and input keys on the keyboard. S402, thevirtual input controller (112) performs the image perspective transformafter identifying the keyboard design drawn by the user. After performingthe image perspective transform, at S404, the virtual input controller (112)generates a key-map using an image processing technique and an OpticalCharacter Recognition (OCR) technique. The OCR technique is used toread a custom characters from printed keyboard and build the key map.

[0093] At S406, the virtual input controller (112) detects the tips ofthe finger (300) using the ML model. At S408, the virtual input controller(112) detects the finger tap using the accelerometer sensor (114a). Based onthe finger tap, the virtual input controller (112) generates the desired keypress using a finger tracking data, such that a character (L) is identified anddisplayed over the electronic device (100). The ML model predicts keytouch based on current accelerometer X, Y, Z and median values. Thenotation "a" of the FIG. 5 illustrates a raw image transformation data. Thenotation "b" of the FIG. 5 illustrates contour detection and OCR data. Thenotation "c" of the FIG. 5 illustrates finger tracking data. Each letter hasdifferent contour.

[0094] FIG. 6 is another example scenario in which the electronicdevice (100) displays corresponding letters on the display (110) bydetecting the location of the finger position on the paper keyboard (400a)and mapping to the keyboard key-map to realize the corresponding keypress on the virtual keyboard, according to an embodiment as disclosedherein.

[0095] Consider, the user of the electronic device (100) draws thekeys specific to his / her need or the keyboard layout of his / her choice, theelectronic device (100) automatically identifies layout (i.e., creating virtualkeyboard). The method can be used to combine with the fingeridentification on the virtual keyboard using the camera (102) and theaccelerometer sensor (114a). The virtual input controller (112) detects thelocation of the finger position and then maps to the keyboard key-map torealize the corresponding key press on the keyboard. Based on themapping, virtual input controller (112) displays the corresponding letter onthe display (110).

[0096] In another example, the electronic device (100) displayscorresponding letters on the display (110) based on a swipe gesture on thekeys (200) of the keyboard of the electronic device as shown in the FIG. 7.In another example, the electronic device (100) displays correspondingletters on the display (110) by typing using the printed keyboard paper asshown in the FIG. 8.

[0097] FIG. 9 is another example scenario in which the electronicdevice (100) displays corresponding letters on the display (110) by typingusing finger tracking, according to an embodiment as disclosed herein. Thefinger tips will be shown on the electronic device (100), while typing, sothat the user of the electronic device (100) can easily track the finger (300)on the keyboard while typing.

[0098] In another example, the electronic device (100) displayscorresponding letters on the display (110) by swiping on the customkeyboard to type as shown in the FIG. 10. In another example, theelectronic device (100) displays corresponding letters on the display (110)by swiping on any surface to type as shown in the FIG. 11.

[0099] FIG. 12 is another example scenario in which the electronicdevice (100) displays corresponding letters on the display (110) byrecognition handwriting on a plastic keyboard (400c) using a pen,according to an embodiment as disclosed herein.

[00100] In another example, the electronic device (100) displayscorresponding letters on the display (110) by recognition handwriting onthe paper keyboard without using the pen. Similarly, the electronic device(100) displays corresponding letters on the display (110) by recognizinghandwriting using the user finger (300) on any surface to type.

[00101] In another example, the electronic device (100) displayscorresponding letters on the display (110) by typing using finger tracking oftwo hands as shown in the FIG. 13. In another example, the electronicdevice (100) displays corresponding letters on the display (110) by typingon a custom keyboard (400d) as shown in the FIG. 14.

[00102] FIG. 15 is an example scenario in which the electronicdevice (100) maps the left hand keyboard layout (1500) to the image of thenormal surface area, according to an embodiment as disclosed herein. Asshown in the FIG. 15, the electronic device (100) detects the left hand ofthe user. After detecting the left hand of the user, the electronic device(100) maps the left hand keyboard layout (1500) to the image of the normalsurface area.

[00103] FIG. 16 is an example scenario in which the electronicdevice (100) maps the right hand keyboard layout (1600) to the image ofthe normal surface area, according to an embodiment as disclosed herein.As shown in the FIG. 16, the electronic device (100) detects the right handof the user. After detecting the right hand of the user, the electronic device(100) maps the right hand keyboard layout to the image of the normalsurface area.

[00104] FIG. 17 is an example scenario in which the electronicdevice (100) maps a full hand keyboard layout (1700) to the image of thenormal surface area, according to an embodiment as disclosed herein. Asshown in the FIG. 17, the electronic device (100) detects both left and righthands of the user. After detecting both left and right hands of the user, theelectronic device (100) maps the full hand keyboard layout (1700) to theimage of the normal surface area, according to an embodiment as disclosedherein.

[00105] FIG. 18 is another example scenario in which the electronicdevice (100) maps the split hand keyboard layout (1802 and 1804) to theimage of the normal surface area, according to an embodiment as disclosedherein. As shown in the FIG. 18, the electronic device (100) detects bothleft and right hands of the user. After detecting both left and right hands ofthe user, the electronic device (100) maps the split hand keyboard layout(1802 and 1804) to the image of the normal surface area, according to anembodiment as disclosed herein.

[00106] FIG. 19 is an example scenario in which the electronicdevice (100) identifies the finger events on the virtual keyboard usingHBC, according to an embodiment as disclosed herein. The HBC uses bodytissue as a transmission medium (1900) to transmit informatics. As shownin the FIG. 19, an ECG signal from chest are modulated into micro-Ampereelectric current and coupled into body by electrodes. The ECG signal isdetected by receiving electrodes on a wrist. The transmitting and receivingelectrodes results in galvanic coupling signal transmission and the signalpropagation is based on ionic current. A small current induced electricpotential can be detected at a receiver by a high-gain differential amplifier.

[00107] FIG. 20 is another example scenario in which the electronicdevice (100) displays corresponding letters on the display (110) by theHBC, according to an embodiment as disclosed herein. As shown in theFIG. 20, the user holds the electronic device (100) in one hand and types onthe surface (2002) using second hand. By using the HBC (2000), theelectronic device (100) will translate finger events on the display (110) andcorresponding letters are displayed on the display (110). This technique isalso applicable to the ultrasonic method.

[00108] In another example, the user will slide fingers to type onthe electronic device (100) while watching the display (110).

[00109] FIG. 21 is an example scenario in which the electronicdevice (100) identifies the finger events on the virtual keyboard using theprism (2102), according to an embodiment as disclosed herein. The prism(2102) with the camera (102) is used to simulate conditions like 2 camera'sto determine the depth map using stereo analysis. The purpose to use theprism (2102) is to capture different angle of perspective. The prism (2102)identifies the finger tap events on the virtual keyboard more accurately.

[00110] FIG. 22 is another example scenario in which the electronicdevice (100) displays corresponding letters on the display (110) using aprediction technique, according to an embodiment as disclosed herein. Theprediction technique can be an existing prediction technique.

[00111] FIG. 23 is an example scenario in which the electronicdevice controls a gallery and a video application in multi window scenarioby detecting a key (2302 and 2304) corresponding to the portion at whichthe tap is performed on the surface area, according to an embodiment asdisclosed herein. The electronic device (100) detects both hands. Afterdetecting the both hands, the finger positions detected and calibrated over functionalities corresponding to each application. Based on the userperforming action using fingers (300) on either hand, the electronic devicecontrols a gallery and a video application in multi window scenario.

[00112] In another example, the electronic device (100) controls thescrolling operation on a music application by detecting the key (2402)corresponding to the portion at which the tap is performed on the surfacearea as shown in the FIG. 24

[00113] In another example, the electronic device (100) controls thetouch pad operation on a music application by detecting a key (2502)corresponding to the portion at which the tap is performed on the surfacearea as shown in the FIG. 25. In another example, the user of the electronicdevice (100) controls buds volume using finger slide while listening tomusic. As shown in the FIG. 26, the electronic device (100) edits links on awebpage by using the virtual input (2602) over the surface.

[00114] The embodiments disclosed herein can be implementedusing at least one software program running on at least one hardwaredevice and performing network management functions to control theelements.

[00115] The foregoing description of the specific embodiments willso fully reveal the general nature of the embodiments herein that others can,by applying current knowledge, readily modify and / or adapt for variousapplications such specific embodiments without departing from the genericconcept, and, therefore, such adaptations and modifications should and areintended to be comprehended within the meaning and range of equivalentsof the disclosed embodiments. It is to be understood that the phraseology orterminology employed herein is for the purpose of description and not oflimitation. Therefore, while the embodiments herein have been described interms of preferred embodiments, those skilled in the art will recognize thatthe embodiments herein can be practiced with modification within the spiritand scope of the embodiments as described herein.

Claims

1. A method for enabling virtual input by an electronic device (100), comprising: measuring, by the electronic device (100), sensor data corresponding to a gesture performed by a user on a surface area (400); detecting, by the electronic device (100), a virtual input event by at least one finger (300) of the user based on the sensor data; determining, by the electronic device (100), at least one portion at which the gesture is performed by the user on the surface area (400) based on the virtual input event; detecting, by the electronic device (100), at least one key (200) corresponding to the at least one portion at which the gesture is performed; and performing, by the electronic device (100), at least one action based on the at least one detected key (200).

2. The method as claimed in claim 1, wherein the surface area (200) is one of a normal surface area and an artificial surface area.

3. The method as claimed in claim 2, wherein the normal surface area is one of a glass surface, a rubber surface, a plastic surface, a wooden surface, and a mattress surface.

4. The method as claimed in claim 2, wherein the artificial surface area is one of a printed keyboard and a handwritten keyboard on the normal surface area.

5. The method as claimed in claim 1, wherein the sensor data comprises at least one of an accelerometer sensor data, a prism data, and an ultrasonic data.

6. The method as claimed in claim 1, wherein the sensor data is obtained from at least one of a camera, a prism, and Human body communication (HBC).

7. The method as claimed in claim 1, wherein the typing event comprises at least one of a tap of the at least one finger (300), a swipe of the at least one finger (300), and a handwriting of the user.

8. The method as claimed in claim 1, wherein measuring, by the electronic device (100), the sensor data corresponding to the gesture performed by the user on the surface area comprises: automatically activating, by the electronic device (100), a camera (102) of the electronic device (100); detecting, by the electronic device (100), whether the surface area (400) corresponds to a normal surface area or an artificial surface area displayed in a field of view of the camera (102); and performing, by the electronic device (100), one of: capturing at least one image of the normal surface area displayed within the field of view of the camera (100) in response to determining that the surface area corresponds to the normal surface, generating a virtual area by mapping a keyboard layout of the electronic device (100) to the at least one image of the normal surface area, detecting the gesture performed by the user in the virtual area, and measuring the sensor data when the gesture is performed in the virtual area, and capturing at least one image of the artificial surface area displayed within the field of view of the camera (102) in response to determining that the surface area corresponds to the artificial surface, generating a keyboard layout provided in the artificial surface by processing the at least one image of the artificial surface area, and measuring the sensor data when the gesture is performed on the normal surface area.

9. The method as claimed in claim 8, wherein mapping the keyboard layout of the electronic device (100) to the at least one image of the normal surface area comprises: detecting, by the electronic device (100), one of a left hand of the user, a right hand of the user and both left and right hands of the user; and performing, by the electronic device (100), one of: mapping a left hand keyboard layout to the at least one image of the normal surface area in response to detecting the left hand of the user, mapping a right hand keyboard layout to the at least one image of the normal surface area in response to detecting the right hand of the user, and mapping a split hand keyboard layout to the at least one image of the normal surface area in response to detecting both the left hand and the right hand of the user.

10. The method as claimed in claim 1, wherein detecting, by the electronic device (100), the typing event by at least one finger of the user based on the sensor data comprises: applying, by the electronic device (100), a machine learning model to detect the at least one finger (300) of the user based on the gesture performed by the user on the surface area; applying, by the electronic device (100), the machine learning model to detect a tip of the at least one finger (300) of the user based on the gesture performed by the user on the surface area (400); and detecting, by the electronic device (100), the typing event performed by the tip of the at least one finger of the user based on the sensor data.

11. The method as claimed in claim 1, wherein detecting, by the electronic device (100), the at least one key (200) corresponding to the at least one portion comprising: detecting, by the electronic device (100), whether the surface area (400) corresponds to a normal surface area or an artificial keyboard; and performing, by the electronic device (100), one of: mapping the at least one portion at which the gesture is performed with at least one key (200) of a keyboard of the electronic device (100) in response to determining that the surface area (400) corresponds to the normal surface area, and detecting the at least one key corresponding to the at least one portion based on the mapping, and mapping the at least one portion at which the gesture is performed to at least one key (200) of a key map provided in the artificial surface area in response to determining that the surface area (400) corresponds to the artificial surface area, and detecting the at least one key (200) corresponding to the at least one portion based on the mapping.

12. An electronic device (100) for enabling virtual input, comprising: a memory (106); a processor (108); at least one sensor (114), coupled to the memory (106) and the processor (108), configured to measure sensor data corresponding to a gesture performed by a user on a surface area (400); and a virtual input controller (112), coupled to the memory (106) and the processor (108), configured to: detect a virtual input event by at least one finger (300) of the user based on the sensor data, determine at least one portion at which the gesture is performed by the user on the surface area (400) based on the virtual input event, detect at least one key (200) corresponding to the at least one portion at which the gesture is performed, and perform at least one action based on the at least one detected key (200).

13. The electronic device (100) as claimed in claim 12, wherein the surface area is one of a normal surface area and an artificial surface area.

14. The electronic device (100) as claimed in claim 13, wherein the normal surface area is one of a glass surface, a rubber surface, a plastic surface, a wooden surface, and a mattress surface.

15. The electronic device (100) as claimed in claim 13, wherein the artificial surface area is one of a printed keyboard and a handwritten keyboard on the normal surface area.

16. The electronic device (100) as claimed in claim 12, wherein the sensor data comprises at least one of an accelerometer sensor data, a prism data, and an ultrasonic data.

17. The electronic device (100) as claimed in claim 12, wherein the sensor data is obtained from at least one of a camera, a prism, and Human body communication (HBC).

18. The electronic device (100) as claimed in claim 12, wherein the typing event comprises at least one of a tap of the at least one finger, a swipe of the at least one finger, and a handwriting of the user.

19. The electronic device (100) as claimed in claim 12, wherein measure accelerometer sensor data corresponding to the gesture performed by the user on the surface area comprising: automatically activate a camera of the electronic device; detect whether the surface area corresponds to a normal surface area or an artificial surface area displayed in a field of view of the camera; and perform one of: capture at least one image of the normal surface area displayed within the field of view of the camera in response to determining that the surface area corresponds to the normal surface, generate a virtual area by mapping a keyboard layout of the electronic device to the at least one image of the normal surface area, detecting the gesture performed by the user in the virtual area, and measure the accelerometer sensor data when the gesture is performed in the virtual area, and capture at least one image of the artificial surface area displayed within the field of view of the camera in response to determine that the surface area corresponds to the artificial surface, generate a keyboard layout provided in the artificial surface by processing the at least one image of the artificial surface area, and measure the sensor data when the gesture is performed on the normal surface area.

20. The electronic device (100) as claimed in claim 19, wherein map the keyboard layout of the electronic device to the at least one image of the normal surface area comprises: detect one of a left hand of the user, a right hand of the user and both left and right hands of the user; and perform one of: map a left hand keyboard layout to the at least one image of the normal surface area in response to detecting the left hand of the user, map a right hand keyboard layout to the at least one image of the normal surface area in response to detecting the right hand of the user, and map a split hand keyboard layout to the at least one image of the normal surface area in response to detecting both the left hand and the right hand of the user.

21. The electronic device (100) as claimed in claim 12, wherein detect the typing event by at least one finger of the user based on the sensor data comprises: apply a machine learning model to detect the at least one finger of the user based on the gesture performed by the user on the surface area, apply a machine learning model to detect a tip of the at least one finger of the user based on the gesture performed by the user on the surface area, and detect the typing event performed by the tip of the at least one finger of the user based on the sensor data.

22. The electronic device (100) as claimed in claim 12, wherein detect at least one key corresponding to the at least one portion comprising: detect whether the surface area corresponds to a normal surface area or an artificial keyboard; and perform one of: map the at least one portion at which the gesture is performed with at least one key of a keyboard of the electronic device in response to determine that the surface area corresponds to the normal surface area, and detect the at least one key corresponding to the at least one portion based on the mapping, and map the at least one portion at which the gesture is performed to at least one key of a key map provided in the artificial surface area in response to determining that the surface area corresponds to the artificial surface area, and detect the at least one key corresponding to the at least one portion based on the mapping.