A System for Assisting a User for Operating Third-Party Mobile Applications
Patent Information
- Application Number
- IN202621013864
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2026-02-09
- Publication Date
- 2026-09-22
- Estimated Expiration
- 2046-02-09
AI Technical Summary
Existing digital assistance systems for mobile applications lack interactivity and context-awareness, leading to confusion and errors for first-time or digitally challenged users, particularly those with visual impairments or limited experience with touchscreens.
A system utilizing OCR, natural language processing, and a guidance generator to provide multimodal guidance, including visual and haptic feedback, tailored to the user's current interface context, to assist in navigating third-party mobile applications.
Enhances user understanding and confidence, reduces errors, and promotes inclusivity by providing clear, timely assistance that adapts to the user's context and interface elements.
Abstract
Description
Field of the invention
[0001] The present invention relates to the mobile applications. Morespecifically, the invention relates to a system for assisting a user for operating thirdpartymobile applications.Background of the invention
[0002] Smartphone adoption has grown rapidly worldwide, yet asignificant portion of users, particularly first-time or older users, struggle withdigital literacy. Despite the large number of tutorials, manuals, and app-learningtools, many of these resources fail to provide guidance that is contextually awareor tailored to the specific interface elements a user encounters in real time. Staticinstructions, whether in the form of printed manuals or video tutorials, often assumea baseline familiarity with smartphone interaction paradigms, leaving new usersconfused or frustrated when confronted with unfamiliar applications.
[0003] Existing solutions in the domain of digital assistance typicallyrely on on-screen prompts, video walkthroughs, or step-by-step instructions thatrequire the user to interpret abstract guidance and manually correlate it with theinterface on their device. Such approaches lack interactivity and cannot adaptdynamically to the user's current context, such as which screen is currently active,which buttons have been pressed, or whether a user has correctly completed a givenaction. This gap between instruction and action creates barriers for users attemptingto learn and navigate new applications independently.
[0004] Furthermore, current tools rarely leverage sensory feedbackmechanisms to guide user attention physically. Humans naturally respond to tactileand multi-sensory cues in addition to visual prompts, yet most digital assistancesystems ignore these modalities. Without such guidance, users may overlookcritical interface elements, press incorrect controls, or become disengaged from thelearning process. This limitation is particularly pronounced for users with visualimpairments, limited experience with touchscreens, or cognitive challenges,highlighting the need for more inclusive and accessible guidance technologies.
[0005] Therefore, there is a need for a system for assisting a user foroperating third-party mobile applications to overcome a few or all drawbacks ofthe existing technologies.Objects of the invention
[0006] An object of the present invention is to provide a system forassisting a user for operating third-party mobile applications.
[0007] Another object of the present invention is to provide a systemfor assisting a user for operating third-party mobile applications that improves theuser's ability to understand and follow operational steps on unfamiliar interfaces.
[0008] Another one object of the present invention is to provide asystem for assisting a user for operating third-party mobile applications thatreduces confusion and errors typically experienced by first-time or digitallychallengedusers.
[0009] Another object of the present invention is to provide a systemfor assisting a user for operating third-party mobile applications that enhances userconfidence by offering clear and timely assistance during application navigation.
[0010] Yet another object of the present invention is to provide asystem for assisting a user for operating third-party mobile applications thatpromotes accessibility and inclusivity for users with limited technologicalfamiliarity.
[0011] One more object of the present invention is to provide a systemfor assisting a user for operating third-party mobile applications that supports asmoother learning experience and reduces reliance on external help or staticinstruction manuals.Summary of the Invention
[0012] According to the present invention, there is provided a systemfor assisting a user for operating third-party mobile applications. The system mayinclude an input module, an optical character recognition (OCR) engine, a natural10language processing engine, a database module, and a guidance generator. Further,the system may include an electronic device having a user interface moduleconfigured to allow a user to capture a screenshot of a third-party mobileapplication and to enter a user query in text or voice form. The input module maybe configured to receive the screenshot of the third-party mobile application andthe user query in text or voice form.
[0013] Further, the OCR engine may be configured to detect userinterface(UI) elements within the received screenshot and to output correspondingbounding-box coordinates. The OCR engine may identify app-specific visualmarkers to determine the identity of the third-party mobile application representedin the screenshot.
[0014] Further, the natural-language processing engine may beconfigured to interpret the user query based on the UI elements detected by theOCR engine. The natural-language processing engine may support multilingualinput and output and constrains translations to match OCR-detected on-screen textfor semantic consistency with the detected text in the screenshot.
[0015] Furthermore, the database module may be configured to storeannotated screenshots, created workflows, and instructional sequences. The storeddata may provide pre-created app guides for generating complete and contextuallyconsistent instructional steps.
[0016] The guidance generator may be configured to access the storeddata from the database module to generate step-wise instructions based on thedetected UI elements. The step-wise instructions are generated on the basis of theinterpretation of the user query and the stored information of the database module.The guidance generator may be configured to link each instruction to acorresponding actionable UI element detected by the OCR engine. The guidancegenerator may provide guidance by highlighting the actionable UI element andactivating a haptic feedback and screen-brightness adjustment to provide tactile andvisual confirmation during user interaction.
[0017] Further, the guidance generator may provide guidance bydimming non-relevant areas, highlighting the actionable UI element detected bythe OCR engine, and activating a device vibration motor to provide haptic feedbackwhen the user interacts with the highlighted actionable UI element.
[0018] Furthermore, the guidance generator may adjust the intensity,duration, or pattern of the haptic-feedback based on a confidence score associatedwith the actionable UI element linked to a corresponding instruction. The guidancegenerator may control the device vibration motor to generate different vibrationpatterns corresponding to different actionable UI elements, thereby enabling theuser to distinguish between multiple instructional steps through tactile cues. Theguidance generator may detect a mismatch between the received screenshot and apreviously generated instruction step and, in response, generates updatedinstructions or requests an updated screenshot. Further, the guidance generator maylink each instructional step textual content of a corresponding actionable UIelement and to bounding-box coordinates generated by the OCR engine.Brief description of drawings
[0019] The advantages and features of the present invention will beunderstood better with reference to the following detailed description and claimstaken in conjunction with the accompanying drawings, wherein like elements areidentified with like symbols, and in which:
[0020] FIG. 1 shows a block diagram of a system for assisting a userfor operating third-party mobile applications in accordance with the presentinvention; and
[0021] FIG. 2 shows flowchart for a method for assisting a user foroperating third-party mobile applications in accordance with the present invention.Detailed description of the invention
[0022] An embodiment of this invention, illustrating its features, willnow be described in detail. The words "comprising," "having," "containing," and"including," and other forms thereof, are intended to be equivalent in meaning andbe open-ended in that an item or items following any one of these words is notmeant to be an exhaustive listing of such item or items, or meant to be limited toonly the listed item or items.
[0023] The present invention relates to a system for assisting a userfor operating third-party mobile applications. The system is configured to supporteffective and intuitive user interaction with mobile interfaces by assisting users inunderstanding and navigating application workflows under a variety of usageconditions. The system is designed to provide consistent support across diverse usergroups, including first-time smartphone users, individuals with limited digitalliteracy, elderly users, and users operating unfamiliar or complex applications. Thepresent invention promotes improved usability, enhanced user confidence, anddependable assistance outcomes while reducing user frustration, minimizingoperational errors, and enabling smoother interaction experiences across a widerange of mobile applications and usage scenarios.
[0024] The terms "first," "second," and the like, herein do not denoteany order, quantity, or importance, but rather are used to distinguish one elementfrom another, and the terms "a" and "an" herein do not denote a limitation ofquantity, but rather denote the presence of at least one of the referenced item.
[0025] The disclosed embodiments are merely exemplary of theinvention, which may be embodied in various forms.
[0026] Referring now to FIG. 1, a system (100) for assisting a userfor operating third-party mobile applications in accordance with the presentinvention is illustrated. Specifically, the system (100) includes an input module(10), an optical character recognition (OCR) engine (20), a natural-languageprocessing engine (30), a database module (40), and a guidance generator (50). Thesystem (100) thereby provides a unified architectural framework for deliveringmultimodal, screen-specific, and hardware-assisted guidance to a user.
[0027] The system (100) includes an electronic device (60)configured to allow a user to provide a screenshot of a third-party mobileapplication and to submit a user query in text or voice form. The electronic device(60) serves as the front-end interaction layer through which the screenshot and theuser query are collected before being forwarded to the input module (10) forsubsequent processing. The electronic device (60) includes a user interface module(60A) configured to present a graphical interface for capturing screenshots andreceiving user queries in text or voice form. It may be obvious to a person skilledin the art that the electronic device (60) may be implemented as a graphicalinterface on a mobile device, a web-based interface, or an integrated software panelwithin a host application.
[0028] The input module (10) is configured to receive a screenshot ofa third-party mobile application and a user query in text or voice form from theelectronic device (60). The input module (10) is configured to receive thescreenshot provided by the user and ensure that the screenshot is captured in aformat suitable for subsequent processing. The input module (10) allows the userto upload the screenshot directly from the user device and supports selection of apreviously stored screenshot so that the system (100) can initiate guidance evenwhen the user is not operating the app in real time. The input module (10) ensuresthat the received screenshot maintains adequate clarity for downstream analysis.
[0029] Further, the input module (10) is configured to receive the userquery in text or voice form, with voice input converted into text to maintainconsistency in processing. The input module (10) performs basic stabilization ofboth the screenshot and the user query so that the downstream components operateon clean and uniform inputs. The input module (10) ensures that the user query isaligned with the screenshot so that the system (100) can correctly interpret theuser's request in relation to the displayed elements. The input module (10) providesthe screenshot to the OCR engine (20), thereby enabling the detection of userinterface(UI) elements contained within the screenshot. The input module (10)provides an audio-output capability for delivering spoken guidance. The inputmodule (10) includes text-to-speech (TTS) functionality that converts the generatedinstructional steps into natural-sounding audio output, enabling hands-free oraccessibility-oriented operation when visual reading of the instructions isimpractical.
[0030] The OCR engine (20) is configured to detect user-interface(UI) elements within the received screenshot and to output correspondingbounding-box coordinates. The OCR engine (20) identifies textual labels, icons,and visually distinct regions so that the system (100) establishes a structuralrepresentation of the displayed screen. The OCR engine (20) performs thisdetection in a manner that preserves spatial relationships between the UI elements,enabling reliable downstream interpretation.
[0031] Further, the OCR engine (20) is configured to processvariations in screen layouts arising from different device models, orientations, anddisplay densities. The OCR engine (20) applies normalization operations to ensurethat the extracted coordinates are consistent across screenshots having differingresolutions or aspect ratios. Further, the OCR engine (20) ensures that the detectedUI elements remain stable even when the screenshot contains overlappingcomponents or partially obscured regions.
[0032] Furthermore, the OCR engine (20) analyzes visualcharacteristics of the detected UI elements to distinguish between actionable UIelements and non-actionable background content. The OCR engine (20) refineseach bounding box by evaluating edge clarity, textual prominence, andiconographic structure, thereby improving the ability of the system (100) tocorrectly target elements when generating guidance. The OCR engine (20) alsosupports repeated detection across multiple screenshots when the user uploads asequence of screens during a multi-step interaction. The OCR engine (20) includesone or more AI (Artificial Intelligence) models, including but not limited to deeplearning-based computer-vision models, convolutional neural networks (CNNs), ortransformer-based visual recognition models, to improve detection of UI elementsand app-specific markers. It may be obvious to a person skilled in the art to useother AI model for enhanced accuracy, adaptability to diverse screen layouts, androbustness to partial occlusion or low-resolution screenshots.
[0033] Moreover, the OCR engine (20) outputs the detected UIelements and coordinates of the bounding-box to the natural-language processingengine (30), ensuring that the natural-language processing engine (30) can interpretthe user query in direct relation to the UI context displayed in the screenshot. TheOCR engine (20) thus provides structured visual information that allows thenatural-language processing engine (30) to understand the user's intent in context.
[0034] The natural-language processing engine (30) is configured tointerpret the user query based on the UI elements detected by the OCR engine (20).The natural-language processing engine (30) receives the detected UI elements andbounding-box coordinates from the OCR engine (20), to associate the user querydirectly with specific visual elements displayed on the screenshot. This ensures thatthe natural-language processing engine (30) understands the user's intent in theprecise context of the current screen layout. The natural-language processingengine (30) includes one or more AI (Artificial Intelligence) models, including butnot limited to deep-learning-based language models, transformer-based models, orsequence-to-sequence neural networks, to derive user intent, map queries toactionable UI elements, and generate context-aware, step-wise instructions. It maybe obvious to a person skilled in the art to use any alternative AI model orreinforcement-learning techniques to improve semantic understanding,multilingual support, and alignment with OCR-detected UI elements.
[0035] Further, the natural-language processing engine (30) analyzesthe user query, whether provided in text or voice form, to extract actionableinstructions, commands, or questions. The natural-language processing engine (30)uses the UI element data provided by the OCR engine (20) to determine which partsof the screenshot are relevant to the user query. The natural-language processingengine (30) ensures that the generated guidance is contextually accurate anddirectly applicable to the interface shown, by aligning semantic information fromthe user query with the visual layout.
[0036] Furthermore, the natural-language processing engine (30)maps the interpreted user intent to the corresponding actionable UI elementsidentified by the OCR engine (20). The natural-language processing engine (30)links each instruction to specific bounding-box coordinates and UI element types,preparing a structured representation of the user's requested actions. Thisstructured data serves as a precise input for the guidance generator (50), enablingstep-wise instruction generation. The natural-language processing engine (30)communicates the interpreted instructions and linked UI element data to thedatabase module (40), which stores pre-created app guides. This allows theguidance generator (50) to access relevant database content and generate step-wiseinstructions that are consistent with both the user query and the known workflowsfor the detected application. The integration of the natural-language processingengine (30) with the database module (40) ensures that each instruction is accurate,contextually relevant, and enriched with pre-created guidance where applicable.
[0037] The database module (40) is configured to store annotatedscreenshots, created workflows, and instructional sequences, the stored dataserving as a knowledge base that provides pre-created app guides useful forgenerating complete and contextually consistent instructional steps. The databasemodule (40) maintains pre-created app guides that correspond to frequently usedthird-party applications, ensuring that relevant instructional content is readilyavailable for generating complete and contextually consistent guidance.
[0038] Further, the database module (40) receives metadata from thenatural-language processing engine (30), including interpreted user queries and thebounding-box coordinates of actionable UI elements detected by the OCR engine(20). The database module (40) uses this metadata to identify the most appropriatepre-created app guide, aligning the stored instructions and annotated screenshotswith the current user context.
[0039] Furthermore, the database module (40) provides the retrievedapp guides to the guidance generator (50), enabling the guidance generator (50) togenerate step-wise instructions that are both contextually relevant and consistentwith known workflows. The database module (40) ensures that each instruction islinked to specific UI elements detected by the OCR engine (20) and interpreted bythe natural-language processing engine (30), enhancing accuracy and usability ofthe guidance. The database module (40) supports updating and expanding thestored knowledge base by including new annotated screenshots, instructionalsequences, or user feedback. The database module (40) continuously updates thepre-created guides, thereby improving the performance of the guidance generator(50) over time and maintaining alignment with evolving third-party applicationinterfaces.
[0040] The guidance generator (50) is configured to access the storeddata from the database module (40) to enhance contextual understanding of the UIelements detected by the OCR engine (20) and interpreted by the natural-languageprocessing engine (30). The guidance generator (50) uses the retrieved app guidesand the user query interpretation to generate step-wise instructions that are relevant,complete, and contextually aligned with the screenshot provided by the inputmodule (10). The guidance generator (50) links each instruction to a correspondingactionable UI element identified by the OCR engine (20). Each instructional stepincludes textual content associated with bounding-box coordinates correspondingto the actionable UI element. The guidance generator (50) includes one or more AI(Artificial Intelligence) models, including but not limited to deep-learning,transformer-based, or reinforcement-learning models, to adaptively linkinstructions to actionable UI elements, prioritize instructional steps, optimize visualhighlighting and haptic feedback, and tailor guidance to the user's interactioncontext. It may be obvious to a person skilled in the art to use any alternative AImodel, or adaptive learning techniques to improve step sequencing, contextawareness,and user-specific guidance efficiency.
[0041] Further, the guidance generator (50) overlays interactivevisual highlights on these actionable elements, providing the user with a hardwareassistedvisual guidance layer that clearly indicates where the next interactionshould occur. The guidance generator (50) is configured to trigger haptic-feedbackactivation and screen-brightness-adjustment actions in synchronization with thehighlighted UI elements. The guidance generator (50) ensures that upon userinteraction with a highlighted actionable UI element, the device vibration motorgenerates tactile confirmation while the actionable region remains visuallyprominent, and non-relevant areas of the screen are dimmed to focus the user'sattention.
[0042] Furthermore, the guidance generator (50) monitors userinteractions and dynamically adjusts instructions based on updated input from thenatural-language processing engine (30) and OCR engine (20). The guidancegenerator (50) ensures that each instructional step remains accurate, contextuallyconsistent, and aligned with both the detected UI elements and the pre-createdguides stored in the database module (40), thereby delivering a seamless andintuitive guidance experience.
[0043] Moreover, the guidance generator (50) is configured tocompute a confidence score for each actionable UI element based on factorsincluding OCR text-recognition certainty, UI-element shape consistency, historicaluser-interaction patterns, and correlation strength between the interpreted userquery and the detected UI element. The guidance generator (50) selectively adjuststhe intensity, duration, or vibration pattern of the haptic-feedback response inproportion to the computed confidence score, such that higher-confidenceactionable elements trigger shorter or sharper tactile confirmations, while lowerconfidence elements trigger longer or more pronounced vibration patterns. Thisconfidence-adaptive haptic strategy improves clarity and reduces ambiguity for theuser during interaction.
[0044] The guidance generator (50) controls the device vibrationmotor to generate distinct vibration patterns corresponding to different actionableUI elements or different steps within a workflow. The vibration patterns mayinclude short pulses, long pulses, multi-pulse sequences, or periodic vibrationbursts, each pattern representing a different UI element type or instructionalcategory. The vibration pattern allows the user to differentiate among multipleguided steps through tactile cues.
[0045] The guidance generator (50) is configured to detect amismatch between a currently received screenshot and a previously generatedinstruction step. The mismatch detection process compares the layout, detected UIelements, and bounding-box coordinates of the new screenshot with the expectedscreen state associated with the previous instruction. When discrepancies exceed apredetermined threshold, such as missing UI elements, unexpected screentransitions, or altered visual structures, the guidance generator (50) automaticallyregenerates updated instructions or prompts the user to capture and upload a newscreenshot. The mismatch-resolution mechanism ensures that the instructional flowremains accurate and relevant to the user's current interface context.
[0046] Further, the system (100) includes an application-identityinference mechanism configured to determine which third-party mobile applicationis represented in the received screenshot. The mechanism operates by analyzing thescreenshot using a combination of visual, textual, and structural cues. Initially, avisual-classification model extracts global image features relating to layoutpatterns, color schemes, iconography, and interface structures, and compares thesefeatures with stored feature profiles of known applications to generate apreliminary identity score. In parallel, the OCR engine (20) provides on-screen textsuch as page titles, menu labels, and app-specific terminology, which are matchedagainst app-specific vocabularies using keyword-matching and similarity-scoringtechniques. The mechanism further performs logo- and app-bar analysis to detectapplication icons or brand markers through template matching or lightweightobject-detection models. When available, metadata embedded within thescreenshot, such as package-name indicators or activity labels, is extracted toprovide an additional high-confidence signal. The system (100) combines thesemultiple signals through a weighted decision model to infer the most probableapplication identity, enabling the database module (40) to retrieve thecorresponding pre-created instructional guide and ensuring that the guidancegenerator (50) produces app-specific, contextually accurate instructional steps.
[0047] In an embodiment, the OCR engine (20) and associated visualanalysiscomponents are configured to identify app-specific visual markers withinthe screenshot in order to determine the identity of the third-party mobileapplication. Such markers may include unique iconography, color themes, brandlogos, page-layout structures, or characteristic typography. By identifying theseapp-specific features, the system (100) determines the target application andenables retrieval of the corresponding pre-created workflows and annotatedscreenshots from the database module (40).
[0048] In another embodiment, the natural-language processingengine (30) supports multilingual input and output, enabling the system (100) tointerpret user queries and generate guidance instructions across multiple languages.The natural-language processing engine (30) includes constrained-translationmechanisms that maintain semantic alignment with OCR-detected on-screen textso that translated output preserves terminology appearing within the screenshot.The constrained-translation mechanisms ensures that translated instructions remainconsistent with the user's interface and avoids generating references to text thatdoes not appear on the displayed screen.
[0049] In an embodiment, the system (100) interacts with externalapplication-store APIs (e.g., Google Play Store and Apple App Store APIs) toretrieve metadata related to the detected third-party mobile application. Themetadata may include the application category, developer information, versionnumber, supported features, and user interface conventions. The retrievedinformation improves the accuracy of query interpretation and instructiongeneration by enabling the system to tailor guidance based on known app attributes.
[0050] By way of a non-limiting example, operation of the system(100) begins when the user interface (60) receives a screenshot and user query, andthe input module (10) forwards the received screenshot to the OCR engine (20),which identifies the user-interface (UI) elements and generates bounding-boxcoordinates representing the detected regions of interest. The input module (10)forwards the received screenshot to the OCR engine (20), which identifies the userinterface(UI) elements and generates bounding-box coordinates representing thedetected regions of interest. Further, the OCR engine (20) outputs the detected UIelements to the natural-language processing engine (30), enabling the naturallanguageprocessing engine (30) to interpret the user query in direct relation to thevisual context of the screenshot. The natural-language processing engine (30) thenproduces a context-aware interpretation of user intent, which is passed to theguidance generator (50) for producing step-wise instructions based on the detectedUI layout and the retrieved information from the database module (40).
[0051] Once the natural-language processing engine (30) delivers theinterpreted user intent, the guidance generator (50) accesses the database module(40) to retrieve pre-created app guides, annotated screenshots, and instructionalsequences corresponding to the identified third-party application. The guidancegenerator (50) combines the interpreted intent with the stored knowledge base andthe bounding-box coordinates generated by the OCR engine (20) to produce a setof structured guidance steps that align with the user's query and the specific UIcontext visible within the uploaded screenshot. Further, the guidance generator (50)links each generated instruction to a corresponding actionable UI element, ensuringthat each step directly reflects an element detected by the OCR engine (20).
[0052] After generating the structured guidance steps, the guidancegenerator (50) provides a hardware-assisted visual guidance layer by overlayinginteractive highlights on the actionable UI elements referenced in each instruction.The guidance generator (50) renders each highlighted region such that it remainsin normal brightness while non-relevant regions of the screenshot are subtlydimmed, thereby improving user focus on the intended UI element. Further, uponuser interaction with a highlighted actionable UI element, the guidance generator(50) activates the device's vibration motor to generate a tactile confirmation pulse,allowing the user to physically sense the completion of each guided action.
[0053] The combined operation of the input module (10), the OCRengine (20), the natural-language processing engine (30), the database module (40),and the guidance generator (50) thereby enables the system (100) to delivermultimodal, context-specific, and hardware-assisted guidance for operating thirdpartymobile applications. This coordinated process ensures that each user receivesclear, step-wise support that is visually highlighted, tactically confirmed, andlinguistically adapted to the specific screen and query under consideration.
[0054] Referring now to Figure 2, a flowchart of a method (200) forproviding guidance for operating a mobile application is illustrated.
[0055] In an aspect, a method (200) for providing guidance foroperating a mobile application is provided in accordance with the present invention.Referring to Figure 2, the method (200) is described in conjunction with the system(100) comprising an input module (10), an optical character recognition (OCR)engine (20), a natural-language processing engine (30), a database module (40),and a guidance generator (50). The method (200) includes the following steps:
[0056] The method (200) starts at step (210).
[0057] At step (220), a screenshot of the third-party mobileapplication and a user query in text or voice form are provided to the input module(10). The input module (10) obtains the screenshot uploaded by the user, sharedthrough a system-level screenshot tool, or captured directly from the host device.The input module (10) collects the screenshot uploaded or captured by the user andaccepts the corresponding user query. The input module (10) performspreprocessing such as format standardization and noise reduction to ensure that thescreenshot is suitable for analysis by downstream components.
[0058] At step (230), the screenshot is analyzed using the OCR engine(20) to detect user-interface (UI) elements and generate bounding-box coordinates.The OCR engine (20) extracts textual content, identifies UI structures such as icons,buttons, and input fields, and assigns bounding-box coordinates to each detectedUI element. The OCR engine (20) thereby produces the spatial and textual UIrepresentation required for downstream interpretation.
[0059] At step (240), the user query is interpreted based on thedetected UI elements using the natural-language processing engine (30). Thenatural-language processing engine (30) determines user intent and correlates theinterpreted intent with the UI context derived from the OCR engine (20). Thenatural-language processing engine (30) thereby produces context-awareparameters that align the user's query with the visual content present in thescreenshot.
[0060] At step (250), an annotated screenshots, workflows, andinstructional sequences stored in the database module (40) is accessed to enhancecontextual understanding of the detected UI elements. The database module (40)provides pre-created app guides, annotated screenshots, created workflows, andinstructional sequences that contribute to improving the completeness andcontextual consistency of the forthcoming instructional steps.
[0061] At step (260), a step-wise instructions linked to actionable UIelements selected from the detected UI elements are generated using the guidancegenerator (50), the generation being based on both the interpreted user query andthe accessed database information. The guidance generator (50) aligns eachinstruction with a specific actionable UI element, creating a structured sequenceappropriate to the user's intent and the detected UI layout.
[0062] At step (270), visual highlights are overlaid on actionable UIelements associated with the instructional steps. Using the bounding-boxcoordinates produced by the OCR engine (20), the guidance generator (50) rendersvisual emphasis on each actionable region while preparing the interface for sensoryfeedback.
[0063] At step (280), a haptic-feedback actions and brightnessadjustmentactions are activated in association with the highlighted actionable UIelements. The guidance generator (50) associates tactile and brightness-controlresponses with each corresponding actionable UI element to enable synchronizedsensory reactions during user interaction.
[0064] At step (290), a tactile and visual confirmation is providedwhen the user interacts with the highlighted actionable UI element. The guidancegenerator (50) triggers the device vibration motor to generate haptic feedback andadjusts screen brightness such that the actionable region remains at normalbrightness while surrounding areas are dimmed.
[0065] The method (200) ends at step (300).
[0066] The present invention has the advantage of providing a system(100) for assisting a user for operating third-party mobile applications capable ofdelivering accurate, context-aware, and user-adaptive assistance across a widerange of application environments. The input module (10) ensures accurate captureof the user's on-screen context, while the OCR engine (20) reliably identifies UIelements needed for precise visual mapping. The natural-language processingengine (30) interprets the user query in relation to the detected UI elements, andthe database module (40) supplies pre-created guides that support consistent andrelevant instruction generation. The guidance generator (50) combines these inputsto produce step-wise instructions linked to actionable UI elements and delivershardware-assisted visual and tactile feedback to aid user interaction. The presentinvention thus provides an integrated guidance solution that improves usability andreduces user effort when operating third-party mobile applications.
[0067] The foregoing descriptions of specific embodiments of thepresent invention have been presented for purposes of illustration and description.They are not intended to be exhaustive or to limit the present invention to theprecise forms disclosed, and obviously, many modifications and variations arepossible in light of the above teaching. The embodiments were chosen anddescribed in order to explain the principles of the present invention best and itspractical application, to thereby enabling others skilled in the art to best utilize thepresent invention and various embodiments with various modifications as aresuited to the particular use contemplated. It is understood that various omission andsubstitutions of equivalents are contemplated as circumstance may suggest orrender expedient, but such are intended to cover the application or implementationwithout departing from the scope of the claims of the present invention.
Claims
1. A system (100) for assisting a user for operating third-party mobile applications, the system (100) comprising: an electronic device (60) having a user interface module (60A) configured to allow a user to capture a screenshot of a third-party mobile application and enter a user query in text or voice form; an input module (10) configured to receive the screenshot of the thirdparty mobile application and the user query in text or voice form; an optical character recognition (OCR) engine (20) configured to detect user-interface (UI) elements within the received screenshot for generating corresponding bounding-box coordinates; a natural-language processing engine (30) configured to interpret the user query based on the UI elements detected by the OCR engine (20); a database module (40) configured to store annotated screenshots, created workflows, and instructional sequences, wherein the stored data provides pre-created app guides for generating complete and contextually consistent instructional steps; a guidance generator (50) configured to access the stored data from the database module (40) to generate step-wise instructions based on the detected UI elements, wherein, the step-wise instructions are generated on the basis of the interpretation of the user query and the stored information of the database module (40); wherein the guidance generator (50) is configured to link each instruction to a corresponding actionable UI element detected by the OCR engine (20), and provides guidance by highlighting the actionable UI element and activating a haptic feedback and screen-brightness adjustment to provide tactile and visual confirmation during user interaction; wherein the OCR engine (20), the natural-language processing engine (30), and the guidance generator (50) include one or more artificial intelligence (AI) models configured to detect UI elements, derive user intent, link each instruction corresponding with actionable UI elements, and provide tactile and visual confirmation during user interaction.
2. The system (100) for assisting a user for operating third-party mobile applications as claimed in claim 1, wherein the natural-language processing engine (30) supports multilingual input and output and matches translated responses with OCR-detected on-screen text for semantic consistency with the detected with the detected UI elements.
3. The system (100) for assisting a user for operating third-party mobile applications as claimed in claim 1, wherein the guidance generator (50) provides guidance by dimming non-relevant areas, highlighting the actionable UI element detected by the OCR engine (20), and activating a device vibration motor to provide haptic feedback when the user interacts with the highlighted actionable UI element.
4. The system (100) for assisting a user for operating third-party mobile applications as claimed in claim 1, wherein the guidance generator (50) adjusts the intensity, duration, or pattern of the haptic-feedback based on a confidence score associated with the actionable UI element linked to a corresponding instruction.
5. The system (100) for assisting a user for operating third-party mobile applications as claimed in claim 1, wherein the guidance generator (50) controls the device vibration motor to generate different vibration patterns corresponding to different actionable UI elements, thereby enabling the user to distinguish between multiple instructional steps through tactile cues.
6. The system (100) for assisting a user for operating third-party mobile applications as claimed in claim 1, wherein the guidance generator (50) detects a mismatch between the received screenshot and a previously generated instruction step and, in response, generates updated instructions or requests an updated screenshot.
7. The system (100) for assisting a user for operating third-party mobile applications as claimed in claim 1, wherein the guidance generator (50) links each instructional step textual content of a corresponding actionable UI element and to bounding-box coordinates generated by the OCR engine (20).
8. A method (200) for assisting a user for operating third-party mobile applications, the method (200) comprising the steps of: receiving a screenshot of the third-party mobile application and a user query in text or voice form using an input module (10); analyzing the screenshot to detect user-interface (UI) elements and generate bounding-box coordinates using an optical character recognition OCR engine (20); interpreting the user query based on the detected UI elements using a natural-language processing engine (30); accessing annotated screenshots, workflows, and instructional sequences stored in a database module (40) to enhance contextual understanding of the detected UI elements; generating step-wise instructions linked to actionable UI elements using a guidance generator (50) based on both the interpreted user query and the accessed database information; highlighting the actionable UI elements associated with the instructional steps; activating a haptic feedback and screen-brightness adjustment associated with the highlighted actionable UI elements; and providing tactile and visual confirmation when the user interacts with a highlighted actionable UI element.
