A system for teaching learning using air canvas and hand gestures
Patent Information
- Application Number
- IN202321016488
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-03-13
- Publication Date
- 2026-10-28
- Estimated Expiration
- 2043-03-13
AI Technical Summary
Existing Human-Computer Interaction (HCI) systems for recognizing hand gestures in digital classrooms are limited by high storage requirements, accuracy dependence on background, cost, and availability of hardware, and lack of personalization for teaching-learning environments.
A system utilizing a machine-learning algorithm with Python programming and hand pose model, combined with object detection, for recognizing hand gestures using a webcam, enabling natural interaction and eliminating the need for additional hardware, with integration into educational platforms for online and physical classrooms.
Enables accurate and intuitive hand gesture recognition, allowing for seamless teaching-learning interactions in digital classrooms, enhancing online education and touchless display capabilities with precise fingertip tracking and gesture identification.
Abstract
Description
TECHNICAL FIELD OF INVENTIONThe present invention related to field of computer and science engineering.More specifically, relates to a system for teaching learning using air canvas and handgestures.BACKGROUND OF THE INVENTIONThe background information herein below relates to the present disclosure butis not necessarily prior art.US20190317988A1 related to a system, methods, and software are disclosedherein for presenting an overlay canvas in response to receiving an editing gesture toexisting text on a canvas. In an implementation, user input is received comprising aninking gesture associated with existing text displayed on a canvas in a user interface.The inking gesture is then determined to comprise any of a plurality of editinggestures. In response to the inking gesture comprising an editing gesture, an overlaycanvas is presented above the canvas in the user interface. Additional user input isreceived comprising inking on the overlay canvas. The inking is then incorporatedinto the existing text on the canvas.US11209975B2 described a system, methods, and software are disclosedherein for facilitating enhanced canvas presentation environments. In animplementation, a user interacts with a touch-enabled display system capable ofdisplaying items on a canvas. In response to a gesture made by the user with respectto an item being displayed, a format-specific interaction model is identified based on aformat associated with the item. A response to the gesture may then be determinedusing the interaction model and the response rendered for display.Digital art is now widely replacing traditional ways of drawing. Digital artdescribes ways to express and share art in digital form. Pen, paper, and writing withchalk and a board are two examples of classic writing methods during Teaching.Utilizing a keyboard, touch-screen surface, digital pen, stylus, hand gloves, etc. arejust a few examples of writing tools used in digital classrooms. With the advancementin technology, the need for the development of natural Human-Computer Interaction(HCl)' systems to replace these traditional systems is increasing rapidly. Followingare some existing HCl approaches that are generally used to recognize palm / finger.There are various drawbacks prior art / existing technologies likes Image-BasedApproach, Instrumental Glove-Based Approach, Simple Colored Glove BasedApproach, LED fitted finger movements.Image-Based Approach: - Image-based approach requires the ML / AI model to betrained on the images dataset in order to recognize the hand (object) movements fromthe input video frame, thus they require large amounts of storage space for storing thisimage dataset and have to explicitly build a DBMS for this dataset. Moreover, theaccuracy of object detection is comparatively less as it significantly depends on thebackground of the input video feed for the detection of hand.Instrumental Glove-Based Approach: - For accurate 3-D input for real-time control,this method requires instrumented glove systems like the P5 Data Glove and theImmersion Cyber glove. These technologies, however, are costly and obtrusive. Theydepend on exoskeletons or include more than a dozen sensors into a glove, both ofwhich might limit movement.Simple Colored Glove Based Approach: - This Approach uses the color pattern of theglove (Glove with a different color at different fingers) to determine the hand'sapproximate position and the correspondence of each hand portion. They use data-driven algorithms to extract from a single camera the configuration of a hand holdingsuch a glove. Although a significant amount of research has used this method forhand-tracking systems for user input, their use has been constrained because thesystem's accuracy depends on the background of the live input feed and assumes thatthe same colored object is not present in the video feed. Additionally, the availabilityof such gloves is a significant issue that must be resolved.LED fitted finger movements: - This approach uses Light Emitting diode for detectionand tracking of the fingertip. The camera tracks the fingertip using the red lightemitted using the LED, but it assumes that there is no red-colored object except theLED light in the field of view of the web camera.There are also a few existing HCl approaches for drawing to image generation,like Auto Draw which is a drawing tool that pairs machine learning with drawingsfrom artists to help everyone create anything visual, fast. But the limitation of the toolis that it cannot be personalized for a Teaching-Learning environment.OBJECTIVE OF THE INVENTIONThe primary object of the present invention is to provides a system forteaching learning using air canvas and hand gestures.Yet another objective of the invention is to enable natural interaction betweenhumans and machines by recognizing and interpreting hand gestures.Yet another objective of the invention is to integrate seamlessly intoeducational platforms, both in physical classrooms and online environments.SUMMARY OF THE INVENTIONAccordingly the following invention provides a system for teaching learningusing air canvas and hand gestures.Herein the present invention hand gesturerecognition implemented with the use of a machine-learning algorithm by usingpython programming, which creates natural interaction between man and machine,which can be applied to classroom Teaching-Learning environment. The proposedsystem uses web camera for capturing a sequence of hand gestures images. The handpose model and object detection configuration implemented to detect palms, fingers,and fingertips.It has a wide scope of operability in the following areas like onlineeducation and advanced touch less displays.BRIEF DESCRIPTION OF DRAWINGThis invention is described by way of example with reference to the followingdrawing where,Figure 1 of sheet 1 illustrates theLine diagram of proposed invention.Where,101 denotes webcam input,102 denotes palm detection,103 denotes fingertip detection,104 denotes drawing on canvas,105 denotes image recognition from canvas drawing,106 denotes integrating with web streaming platform,107 denotes architecture of Linux.Figure 2 of sheet 2 illustrates theblock diagram of proposed invention.Where,201 denotes teacher draw the gestures on air canvas,202 denotes fetch the image from the database for identified gestures,203 denotes preparation of lecture by the teacher.DETAILED DESCRIPTION OF THE INVENTIONAs used in the description herein and throughout the claims that follow, themeaning of "a," "an," and "the" includes plural reference unless the contextclearly dictates otherwise. Also, as used in the description herein, the meaning of "in"includes "in" and "on" unless the context clearly dictates otherwise.The present invention is related to a system for teaching learning using aircanvas and hand gestures. Herein the present invention hand gesture recognitionimplemented with the use of a machine-learning algorithm by using pythonprogramming, which creates natural interaction between man and machine, which canbe applied to classroom Teaching-Learning environment.The proposed system uses web camera for capturing a sequence of handgestures images. The hand pose model and object detection configurationimplemented to detect palms, fingers, and fingertips. It has a wide scope of operabilityin the following areas like online education and advanced touch less displays.In order to control the cursor using gestures and hand control, the system usesa finger instead of the additional hardware requirement. Visual recognition of handgestures combined with camera inputs is used in this process.Using live video images and motion tracking, a virtual mouse can beimplemented that recognizes fingertip movements and hand gestures. The tracking ofhand gestures is triggered by the detection of contours around the hand and theformation of convex hulls around it.Based on the contour and hull area ratios, the system can extract features ofhands and can accurately track the fingertips by using these details and drawing themon input from the webcam. After drawing on air canvas system can identify thegesture and accordingly the relevant course content can be displayed on screen.The preferred embodiment the present system will perform the following step:Hand Gesture tracking using Object Detection.Finger Detection using Hand Pose Model.Implementation of the on-screen cursor.Gesture Identification and fetching course content from Database.In the present following technology are implemented:Hand Pose ModelObject detection using Tensor flowPython (Open Source Computer Vision Library)The Functional Specifications of proposed invention are as follows:The present system uses Web Camera for capturing a sequence of hand gesturesimages. The Hand Pose Model and Object Detection algorithms detect palms, fingers,and fingertips.The system produces a drawing canvas in real-time in response to live input in a webbased application and displays the same on the display screen.Fingertip Recognition Model precisely tracks fingertip's movement / gesture and theresulting output is displayed on the canvas so that one can verify whether the correctgesture is detected by the system or not so that the Gesture Identification applicationcan use it.The system will display identified gesture's relevant text / figures / diagrams / imageson the display screen.Further figure 2 shows the block diagram for present invention. Along withthe preparation for the lecture teacher prepares a table containing the mapping ofGesture-to-Image and uploads / saves the same into the database. During the lecturewhenever the teacher performs a particular gesture, the system verifies the gesture.If the gesture is found, in the registered list the figure is displayed on the AirCanvas. If the gesture is not found Error message "Unregistered Gesture" will beshown. Then system fetches the Gesture-to-Image mapping information from thedatabase and locates the system drive path where the Figure / Image is stored on thesystem. Finally, the Figure / Image will be displayed by system on the screen.ADVANTAGES OF THE INVENTIONThe present invention can bring ease of communication. It has a wide scope ofoperability in the following areas:Online Education: The system can be deployed with various online meet platforms,to ease the presenter. The presenter / host can directly use the camera and fingertippositions to draw on the camera feed, or the presentation.Presentations: The user can highlight, point, and interact with the presentation usinghis / her hands only. This provides freedom of movement.Whiteboard: The user can use the whiteboard and / or the direct video output as awhiteboard. This board will take the fingertips, and it will map them on the modifiedvideo feed of the user.Advanced Touch Less Displays: This technology can be further developed intoinnovation of touch-free displays. These displays will use teacher's finger as a pointerand perform the corresponding actions accordingly.While various embodiments of the present disclosure have been illustrated anddescribed herein, it will be clear that the disclosure is not limited to theseembodiments only. Numerous modifications, changes, variations, substitutions, andequivalents will be apparent to those skilled in the art, without departing from thespirit and scope of the disclosure, as described in the claims.
Claims
1. A system for teaching learning using air canvas and hand gestures, comprising of; a web camera configured to capture sequences of hand gestures; a machine-learning configuration implemented using Python for hand gesture recognition and interaction between users and a computing system; a hand pose models and object detection configurations for detecting and interpreting hand gestures, including palms, fingers, and fingertips; a drawing canvas generated in real-time based on live input displayed on a screen or web-based application; a fingertip recognition models accurately tracking fingertip movements and displaying outputs on the canvas; a database containing mappings of gestures to course content; wherein the system enables natural interaction between users and machines, facilitating teaching-learning environments in physical classrooms and online platforms, and providing advantages in online education, presentations, whiteboard interactions, and touchless displays.
