Execution of commands generated on device by user

Abstract

The invention relates to a method for executing a command generated within a framework of an active application via at least one input peripheral of an electronic device, the method is implemented by another application running in the background and includes: obtaining a character string of at least one character displayed on a display peripheral of the electronic device in a graphical environment of an active application and generated from at least one data from an input peripheral; determining whether the character string constitutes a command from a user of the electronic device based at least on the character string and a context of the active application; and when the determination is positive, deleting the character string on the display peripheral device and executing the command.

Description

Technical Field

[0001] The present invention relates to the field of computer applications, and more particularly to command execution services provided by, for example, personal computers. Prior art

[0002] Computer terminals (personal computers, tablets, smartphones, etc.) offer users many applications (messaging applications, word processing, spreadsheets, search engines, calculators, etc.).

[0003] In the current state of the art, each of these applications runs in its own environment, usually in a window, and information entered in these environments is interpreted and processed within the context of these applications.

[0004] For example, information entered by a user while using a word processor is interpreted only within the context of the word processor.

[0005] The present invention provides a solution that overcomes these limitations. Summary of the invention

[0006] The present invention improves the prior art and for this purpose proposes a method for executing a command generated via at least one input peripheral of an electronic device within the framework of an active application, the method being implemented by another application running in the background.

[0007] According to the present invention, the method comprises:

[0008] - obtaining a string of at least one character displayed on a display peripheral of said electronic device in a graphical environment of an active application and generated by at least one data from said at least one input peripheral;

[0009] - determining whether the string constitutes a command from a user of the electronic device, the determination being based on the string;

[0010] And when the determination is positive:

[0011] - deleting said character string on said display peripheral device; and

[0012] - Execute said command.

[0013] In particular, in some embodiments, the method comprises:

[0014] - obtaining a string of at least one character displayed on a display peripheral of said electronic device in a graphical environment of an active application and generated by at least one data from said at least one input peripheral;

[0015] - determining whether the string constitutes a command from a user of the electronic device, the determination being based on the string and also based on:

[0016] the activity application; and / or

[0017] the date on which the string was obtained; and / or

[0018] The location where the string was obtained;

[0019] And when the determination is positive:

[0020] - deleting said character string on said display peripheral device; and

[0021] - Execute said command.

[0022] According to at least one embodiment, the method comprises:

[0023] - a step of obtaining a string of at least one character displayed on a display peripheral of said electronic device in a graphical environment of an active application and generated by at least one data coming from said at least one input peripheral;

[0024] - a step of determining, based at least on the character string and the context of the active application, whether the character string constitutes a command from a user of the electronic device; and when the determination is positive;

[0025] - a step of deleting the string on the display peripheral device of the electronic device; and

[0026] - Steps to execute said command.

[0027] Advantageously, the execution method makes it possible to obtain a character string generated by a user in an active application, detect whether this character string constitutes a command outside the framework of the active application, and if so, execute it and delete the text corresponding to this command on the display peripheral.

[0028] Thus, with the present invention, a user can type a command that cannot be interpreted within the framework of the active application without leaving the active application, the command being detected and executed in the background by another application. For example, a command intended to control a home automation device, such as turning on a light, can be detected and executed in a word processing application, and the text corresponding to the command is also deleted from the text currently being edited in the word processing application.

[0029] The graphical environment of a software application refers to any display area allocated to the application on a display peripheral device (such as a screen) coupled to an electronic device on which the application is at least partially running. In particular, windows of a window system, pop-up bubbles, electronic forms and information areas constitute examples of graphical application environments within the meaning of the present invention.

[0030] The graphical environment of the active application is usually specified by the "active window".

[0031] An input peripheral is any device capable of interpreting user actions in the form of commands, functions or instructions, which allows characters (alphanumeric, punctuation, etc.) to be printed, stored or transmitted. Input peripherals are for example keyboards, screens and / or touch surfaces, mice, microphones (via voice recognition software), etc. For simplicity, it can then be said that a character string is entered, regardless of the input peripheral or the method used for its generation.

[0032] A display peripheral is any device, such as a screen or projector, that is capable of graphically presenting multimedia content (text, graphical interfaces, images, videos, animations, clickable links, buttons, thumbnails, etc.).

[0033] An electronic device is any device that is capable of managing at least display peripherals and / or input peripherals (personal computer, smartphone, electronic tablet, television, onboard computer of a car, connected object, etc.).

[0034] According to at least one embodiment, the method as described above includes continuously storing characters input by a user in a buffer of a memory reserved for an active application.

[0035] According to at least one embodiment, the determining takes into account the type of input area for the character string.

[0036] According to at least one embodiment, the method as described above is characterized in that it comprises:

[0037] - a step of determining a display position of said character string on said display peripheral device; and

[0038] - The step of displaying, on said display peripheral and in the vicinity of said location, the result of the step of executing said command.

[0039] In other words, in at least this embodiment, the text of the user-generated command is replaced by the results of executing the command.

[0040] For example, if a user enters a string in a text editor ("Convert 32 degrees Fahrenheit to Celsius"), the string is automatically detected in the background, interpreted as a temperature conversion command, and the string is replaced by the conversion result "0℃".

[0041] For example, if a user enters a string in the graphics rendering software ("insert last image received by email"), the string is automatically detected by the application in the background, interpreted as a command, and the string is automatically replaced by the last image received by the user's messaging application.

[0042] For example, in at least one embodiment, where the messaging application provides an application programming interface (API), the method may query the API to obtain the last image received via email. Alternatively, the messaging application may record any image received via email and the date the image was received in a memory known to the method.

[0043] The position where the resulting string has to be inserted can be determined, for example, by storing, for example, the coordinates of the mouse or cursor each time a string is newly inserted in the active application.

[0044] As a variant, the position may be determined within an image representing a graphical interface of the electronic device.

[0045] Thus, when the correspondence between the last word(s) entered by the user (and representing a command) in the active window and the text element generated by optical character recognition is found, the method can determine the position of the last word(s) entered within the image by recovering, for example, the position of the last character of the group of words. Once the location of the position / position is determined, the method can display the result of the command execution at that position.

[0046] The location may, for example, correspond to coordinates (in pixels, centimeters, etc.) that are interpretable by a display peripheral of the electronic device.

[0047] Therefore, according to at least one embodiment, the method as described above is characterized in that it includes the following steps: searching and, when the search is positive, detecting the string in at least a second string, the second string being obtained by an optical character recognition step applied to all or part of an image representing the content displayed by the display peripheral device.

[0048] According to at least one embodiment of the invention, the method as described above is characterized in that the optical character recognition is applied to a portion of the image, the coordinates of which correspond to the coordinates of the graphical environment of the active application (eg active window) displayed by the display device.

[0049] This embodiment makes it possible to restrict the execution of optical character recognition to a region / portion representing the content of an image displayed by a display peripheral of the electronic device corresponding to the graphical environment of the active application. This embodiment makes it possible to optimize the use of resources (memory, processor, etc.) required to execute the method.

[0050] According to at least one embodiment, the method as described above is characterized in that the method is executed once at least one special character is detected. For example, before any command, the user may be required to input a command with a fixed string, which may be parameterized as, for example, "##".

[0051] As a variant, according to at least one embodiment, the method as described above is characterized in that it is executed as soon as the user completes a word (a space or punctuation mark is detected) or pauses during input (at least a first duration, which can be parameterized, for example of the order of a few seconds or even tens of seconds (such as 2, 3, 5, 10, 15 or 20 seconds)).

[0052] As a variant, according to at least one embodiment, the method as described above is characterized in that it is performed at regular intervals, for example of a duration of about a few seconds or even tens of seconds, such as 2, 3, 5, 10, 15 or 20 seconds. This duration can be parameterized, for example.

[0053] Such an embodiment makes it possible to adjust the processing time of the command by varying the duration of the interval.

[0054] According to at least one embodiment, the determining step comprises a semantic analysis of the string, and the executing step comprises processing, by at least one agent, a result of the semantic analysis.

[0055] The implementation makes it possible to install different agents to handle different types of commands (numerical computing, home automation, accounting, business applications, etc.). The agents can be implemented, for example, in the form of "plug-ins" and / or additional modules. Depending on the embodiment, the agents can be dedicated to one or more applications.

[0056] In at least one embodiment, executing the method includes a pre-activation step via an input interface of the device.

[0057] In at least one embodiment, the execution method is performed when:

[0058] - upon detection of at least a first character (such as a predetermined character); and / or

[0059] - upon detection of the end of a word; and / or

[0060] - upon detecting a pause in the generation of said character.

[0061] The invention also relates to a device for executing commands generated within the framework of an active application via at least one input peripheral of an electronic device.

[0062] The device can be configured to implement the method of the present application in any one of the embodiments of the present application.

[0063] Therefore, according to at least one embodiment, the device comprises:

[0064] - means for obtaining, in a graphical environment of an active application, a string of at least one character displayed on a display peripheral of said electronic device and generated by at least one data from said at least one input peripheral;

[0065] - a determination module configured to determine whether the string constitutes a command from a user of the electronic device, the determination being based on the string;

[0066] - a module for deleting the string on the display peripheral device of the electronic device; and

[0067] - A module for executing said command.

[0068] More specifically, according to at least one embodiment, the device includes:

[0069] - means for obtaining, in a graphical environment of an active application, a string of at least one character displayed on a display peripheral of said electronic device and generated by at least one data from said at least one input peripheral;

[0070] - a determination module configured to determine, based at least on the character string and the context of the active application, whether the character string constitutes a command from a user of the electronic device;

[0071] - a module for deleting the string on the display peripheral device of the electronic device; and

[0072] - A module for executing said command.

[0073] According to at least one embodiment, the device comprises:

[0074] - means for obtaining, in a graphical environment of an active application, a string of at least one character displayed on a display peripheral of said electronic device and generated by at least one data from said at least one input peripheral;

[0075] a determination module configured to determine whether the string constitutes a command from a user of the electronic device, the determination being based on the string and also based on:

[0076] the activity application; and / or

[0077] the date on which the string was obtained; and / or

[0078] The location where the string was obtained;

[0079] - a module for deleting the string on the display peripheral device of the electronic device; and

[0080] - A module for executing said command.

[0081] The term module may correspond to both software components and hardware components or to a collection of hardware and software components, the software components themselves corresponding to one or more computer programs or subroutines, or more generally to any element of a program capable of implementing a function or set of functions as described for the module in question. In the same way, a hardware component corresponds to any element in a hardware set capable of implementing a function or set of functions for the module in question (integrated circuit, smart card, memory card, etc.).

[0082] The invention also relates to a computer program comprising instructions, which, when executed by a processor, are used to implement the above method according to any one of the above specific embodiments. The method can be implemented in various ways, in particular in wired form or in software form. The program can use any programming language and can be in the form of source code, object code or intermediate code between source code and object code, such as in partially compiled form or in any other desired form.

[0083] The invention also relates to a system for executing commands generated within the framework of an active application via at least one input peripheral of an electronic device.

[0084] The system can be configured to implement the method of the present application in any one of the embodiments of the present application.

[0085] Therefore, according to at least one embodiment, the system includes:

[0086] - means for obtaining, in the graphical environment of an active application, a string of at least one character displayed on a display peripheral of said electronic device and generated by at least one data coming from said at least one input peripheral;

[0087] - a determination device configured to determine whether the string constitutes a command from a user of the electronic device, the determination being based on the string;

[0088] - a device for deleting the string on the display peripheral device of the electronic device; and

[0089] - means for executing said commands.

[0090] More specifically, according to at least one embodiment, the system is characterized in that it comprises:

[0091] - means for obtaining, in the graphical environment of an active application, a string of at least one character displayed on a display peripheral of said electronic device and generated by at least one data coming from said at least one input peripheral;

[0092] - a determination device configured to determine, based at least on the character string and the context of the active application, whether the character string constitutes a command from a user of the electronic device;

[0093] - a device for deleting the string on the display peripheral device of the electronic device; and

[0094] - means for executing said commands.

[0095] According to at least one embodiment, the system includes:

[0096] - means for obtaining, in the graphical environment of an active application, a string of at least one character displayed on a display peripheral of said electronic device and generated by at least one data coming from said at least one input peripheral;

[0097] a determination device configured to determine whether the string constitutes a command from a user of the electronic device, the determination being based on the string and also based on:

[0098] the activity application; and / or

[0099] the date on which the string was obtained; and / or

[0100] The location where the string was obtained;

[0101] - a device for deleting the string on the display peripheral device of the electronic device; and

[0102] - means for executing said commands.

[0103] The present invention further relates to a computer-readable recording medium or information medium, and includes the instructions of a computer program as described above. The above-mentioned recording medium can be any entity or device capable of storing a program. For example, the medium can include a storage component, such as a ROM, such as a CD ROM or a microelectronic circuit ROM, or a magnetic recording component, such as a hard disk. On the other hand, the recording medium can correspond to a transmissible medium, such as an electrical signal or an optical signal, which can be transmitted via a cable or an optical cable, by radio or by other means. Programs according to the present invention can be downloaded particularly on an Internet-type network.

[0104] Alternatively, the recording medium may correspond to an integrated circuit in which the program is incorporated, the circuit being suitable for executing or for executing the method in question.

[0105] The execution device, the execution system and the computer program have similar characteristics and advantages as described above with respect to the execution method in all embodiments thereof.

[0106] Different embodiments of the execution method, execution device or execution system described above or below may be combined. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Other characteristics and advantages of the invention will become more apparent from a reading of the following description of particular embodiments given as simple illustrative and non-limiting examples, and of the accompanying drawings, in which:

[0108] [ Figure 1 ] Figure 1 shows an exemplary implementation environment of the present invention according to a specific embodiment of the present invention,

[0109] [ Figure 2A ] Figure 2A shows the architecture of a device suitable for implementing the execution method according to a specific embodiment of the present invention,

[0110] [ Figure 2B ] Figure 2B shows the architecture of a device suitable for implementing the execution method according to another specific embodiment of the present invention,

[0111] [ Figure 3 ] Figure 3 An exemplary implementation of the present invention is shown;

[0112] [ Figure 4 ] Figure 4 An exemplary implementation of the present invention is shown;

[0113] [ Figure 5 ] Figure 5 The main steps of the implementation method according to a specific embodiment of the present invention are shown. DETAILED DESCRIPTION

[0114] Figure 1 An exemplary implementation environment of the present invention according to at least one embodiment is shown. Figure 1 The environment shown in FIG. 1 includes at least one electronic device 101 , which integrates a device for executing commands and is capable of implementing the method for executing commands according to the present invention.

[0115] According to other embodiments, the execution device may be located in a network and / or distributed on one or more computer machines such as computers, terminals or servers.

[0116] The electronic device 101 is, for example, a terminal of the smartphone type, a tablet, a connected television, a connected object, an onboard computer of a car, a personal computer or any other terminal capable of visually and / or acoustically rendering multimedia content. One or more graphics / display or sound 105 rendering peripherals may be connected or included by the electronic device 101 (connected in a wired (via VGA, HDMI, USB cable, etc.) or wireless (WiFi®, Bluetooth®, etc.) manner). This or these rendering peripherals may be, for example, a screen, a video projector, a speaker, etc.

[0117] According to one embodiment of the present invention, (one or more) graphics or sound rendering peripheral devices may be connected to the electronic device 101 via the network 102 .

[0118] Likewise, one or more input peripherals 103a, 103b may be connected or included by the electronic device 101 (either wired (via VGA, HDMI, USB cable, etc.) or wireless (WiFi®, Bluetooth®, etc.)). The input peripheral(s) may be, for example, a keyboard, a mouse, a touch surface, a camera 104, a microphone, or any other peripheral capable of providing interaction data from a user of the electronic device 101.

[0119] Figure 2A and Figure 2B Each shows a device S configured to implement an acquisition method according to at least one embodiment of the present invention. The device S has the conventional architecture of a computer and comprises in particular an operating system OS, a memory MEM, a processing unit UT equipped with, for example, a processor PROC and driven by a computer program PG stored in the memory MEM, an acquisition module MOD_OBT, an action management module MOD_GA and a determination module MOD_DET. The computer program PG comprises a module for implementing support functions as described later when the program is executed by the processor PROC. Figure 5 Instructions for executing the steps of a method.

[0120] At initialization, the code instructions of the computer program PG are loaded into a memory, for example, before being executed by the processor PROC. The processor PROC of the processing unit UT specifically implements the instructions according to the present application (for example, Figure 5 ) any one of the embodiments described and the steps of the execution method according to the instructions of the computer program PG.

[0121] In at least one embodiment, the determination module MOD_DET is configured to interface with a plurality of agents. Figure 2A and Figure 2B Only three of these agents are represented.

[0122] exist Figure 2A In at least one embodiment shown, the determination module MOD_DET includes a client CLT_KNOW and a knowledge server SRV_KNOW included in the device S. The device S may also include at least some agents.

[0123] exist Figure 2B In at least one embodiment shown, the client CLT_KNOW and the knowledge server SRV_KNOW are connected via a communication network.

[0124] The acquisition module MOD_OBT is configured to obtain at least one string of characters from data (sound, movement, click, press (long or short), etc.) generated by at least one user at an input peripheral device (103a (touchpad), 103b (keyboard)) of the electronic device 101. These data are interpreted and converted into characters. This is, for example, the following situation: when pressing a key of the keyboard 103b of the electronic device 101, during an action (movement, click) performed on the touch surface 103a, during the text transcription of the speech of the sign language captured by the camera 104 of the electronic device 101, or during the text transcription of the audio speech captured by the microphone (not represented) of the electronic device 101.

[0125] In at least some embodiments of the method, the acquisition module MOD_OBT can subscribe to the operating system OS in order to be notified of at least some system events. For example, the acquisition module MOD_OBT receives from the operating system OS:

[0126] - an identifier of the active application during (eg, each) change of the active application; and

[0127] - An event representing at least some user action (mouse click, key input on the keyboard, etc.) used to generate input data.

[0128] In at least some embodiments of the method, the memory MEM includes at least one buffer BUFF (eg, a single buffer) storing characters continuously generated by user input in the window of the active application, which are obtained by the obtaining module MOD_OBT through notification of the operating system OS.

[0129] For example, in Figure 3 In the example shown, assume that the user:

[0130] i / Enter the string "Dear" in the messaging application MSG with the identifier ID_MSG (in Figure 3 , shown by the string "Cccc" in the figure); then

[0131] ii / Switch to another application, for example a text processing program TTXT with the identifier ID_TTXT, to enter the string "fish" in the graphic object OG (in Figure 3 Then, the string "translate trout into English" is entered in the text area ZT (indicated by the string "ppppppp" in Figure 3 , represented by the string "Tttttttt tttttt ea aaaaaaa"); then

[0132] iii / Switch back to the messaging application MSG to Figure 3 ), the buffer BUFF includes the string "Dearfishtranslate troutinto English Sir".

[0133] In another embodiment of the method, the memory MEM comprises a buffer per application and / or per application window to continuously store characters entered by the user in the window of the application while the application is active.

[0134] Therefore, by reconsidering Figure 3 By way of example, the memory MEM may include:

[0135] -First buffer BUFF MSG , which includes a character string "Dear Sir" corresponding to characters input by the user in two stages when the messaging application MSG is active; and

[0136] -Second buffer BUFF TTXT , which includes the character string "fishtranslate trout to English" corresponding to the characters entered by the user when the word processing TTXT is the active application.

[0137] In the embodiments described herein, upon detecting the end of a word in a string, such as by detecting a space, punctuation, or a pause greater than a first duration in the user input, the method determines the context of the active application in which the string was entered. The determined context may, for example, allow the method to determine (i) whether the word constitutes a command and (ii) an action to be performed in order to execute the command.

[0138] In fact, the same string may or may not be interpreted as a command, depending on the active application that generates the string. For example, the word "holidays" may be interpreted as a command to access a holiday-related website in the user's profile (e.g., history) only if the word "holidays" is entered into the URL of a web page.

[0139] Furthermore, the same command may be handled differently depending on the context of the active application. For example:

[0140] - based on graphical objects or words detected in the vicinity of the string; and / or

[0141] - Based on the location and / or date of use of the electronic device 101 when the string was generated.

[0142] In at least some detailed embodiments, the device S may include an optical character recognition module MOD_OCR capable of performing optical character recognition on an image (image capture) representing content displayed by a display peripheral device (105) of the electronic device 101 and obtaining at least a second string TXT2 as a result of optical character recognition.

[0143] It should be noted that the image is generated, for example, via software capable of performing the capture of the graphical content of the screen 105 of the electronic device 101 .

[0144] Therefore, by reconsidering the above Figure 3 In the example in FIG. 1 , three strings TXT2 can be obtained by optical character recognition applied to an image representing the content displayed on the screen 105. 1 、TXT2 2 and TXT2 3 :

[0145] -TXT2 1 : "Dear Sir"; and

[0146] -TXT2 2 : "fish"; and

[0147] -TXT2 3 : "translate trout into English".

[0148] In at least some detailed embodiments, the device S may also include a search module MOD_SEARCH capable of searching and detecting a string (e.g., one or more words) obtained via the acquisition module MOD_OBT within the string TXT2 obtained as a result of optical character recognition. For example, the search may be performed once the module MOD_OBT detects the end of a word in the string.

[0149] In at least some embodiments, optical character recognition may be optional, but as shown in the example of text processing TTXT, it can help distinguish several character strings entered by a user in different input areas of the same application ("fish", "translate trout into English").

[0150] In at least some detailed embodiments, the optical character recognition module MOD_OCR is configured to determine that the character string is located near a graphic object (eg, a graphic object or a specific text).

[0151] For example, in Figure 3 In the case of the image, the optical character recognition module MOD_OCR can determine that the string "fish" is located in the rectangle.

[0152] For example, in Figure 4 In the case of an image of Figure 4 ) is located in an oval shape near a magnifying glass of a search engine application SRCHENG with an identifier ID_SRCHENG.

[0153] In at least one embodiment, the input area type may be determined, for example, by implementing a pre-learning mechanism based on annotated graphical interface images. In particular, some embodiments may use a method for determining a URL input area, such as a method that operates based on the recognition of the (elongated) shape and position (usually at the top) of such an input area within the graphical environment of an application presenting such an area.

[0154] In at least some embodiments, the optical character recognition module MOD_OCR is configured to determine a position POS of a character string in an image representing content displayed on the screen 105 .

[0155] In at least some embodiments, device S may include a determination module MOD_DET configured to determine whether the string obtained by the obtaining module MOD_OBT constitutes a command from a user of electronic device 101 based at least on the context of the application of active electronic device 101 when the string is obtained.

[0156] In at least some embodiments, the determination module MOD_DET may include a client CLT_KNOW, for example cooperating with a knowledge server SRV_KNOW (eg implemented in the form of a web service, a library or a database).

[0157] The context of an activity application may specifically include:

[0158] - Identifier of the active application ID_APP;

[0159] - the type of input area in which the string was detected;

[0160] - information relating to one or more graphic and / or text objects detected by the module MOD_OCR and situated in the vicinity of the character string;

[0161] - The date and / or location at which the electronic device 101 was used when the character string was obtained.

[0162] The client CLT_KNOW and the knowledge server SRV_KNOW can communicate according to at least one protocol. For example, the protocol makes it possible to characterize a string, an active application identifier, a type of input area, a date or place of use of an electronic device, a graphic object including a string or located near a string, and / or a text located near a string. Therefore, the protocol cited as a simple illustrative example may include:

[0163] - The keyword "TXT" to introduce the string;

[0164] -Keyword "ID_APP" to introduce the activity application identifier

[0165] - the keywords "DATE" and "GPS" to introduce the date or location of use of the electronic device, respectively;

[0166] - The keyword "ZONE_TYPE" is used to introduce the type of the input zone in which the string was detected. For example, the keyword WEB_URL_SEARCH can be used to specify that the string was detected in the URL input zone;

[0167] - the keyword "IN" for introducing a graphic object containing a string; and

[0168] - The keyword "PROX" is used to introduce a graphic object or text located near a character string.

[0169] For example, by reconsidering Figure 3 and Figure 4 As an example, a client CLT_KNOW can use the following format in a message in a knowledge server SRV_KNOW:

[0170] When the acquisition module MOD_OBT detects the end of the word "Dear", MSG 1={TXT=“Dear”, ID_APP=ID_MSG, DATE=20220715, GPS=48.856614, 2.3522219},

[0171] When the acquisition module MOD_OBT detects the end of the word "Sir", MSG 2 ={TXT=“Dear Sir”, ID_APP=ID_MSG, DATE=20220715, GPS=48.856614, 2.3522219},

[0172] When the acquisition module MOD_OBT detects the end of the word "fish", MSG 3 ={TXT="fish",ID_APP=ID_TTXT,IN=rectangle},

[0173] When the acquisition module MOD_OBT detects the end of the word "English", MSG 4 ={TXT=“translatetrout into English”,ID_APP=ID_TTXT},

[0174] When the acquisition module MOD_OBT detects the end of the word "Euros", MSG 5 ={ZONE_TYPE:WEB_URL_SEARCH=“buy stock X pour 20 Euros”, ID_APP=ID_SRCHENG, IN=oblong, PROX=magnifying glass}.

[0175] The message sent after detecting the end of the words "translate", "trout", "into", "buy", "stock", "X", "to", "20" is not detailed, but is in the same format.

[0176] In one or more embodiments, the knowledge server SRV_KNOW includes a semantic analysis module for analyzing messages received from the client CLT_KNOW based on these keywords (called "tags" in English) and interpretation rules.

[0177] Upon receipt of such a message, and after semantic analysis, the knowledge server SRV_KNOW is configured to determine whether the string included in the message constitutes a command, and if so, to execute the command.

[0178] As previously described, in some embodiments, determining whether a string of characters constitutes a command may depend on the context of the string of characters and the active application.

[0179] Similarly, as previously noted, when a string of characters is determined to constitute a command, in some embodiments, the actions implemented to execute the command may depend on the context of the string of characters and the active application.

[0180] In one or more embodiments, the semantic analysis module of the server SRV_KNOW therefore analyzes the received message to determine its meaning based on the string (keyword TXT) and / or the context of the active application and produces semantic results in the form of rules interpretable by one or more third-party agents. Figure 2A and Figure 2B In the example shown, only 3 agents are represented, but in practice the number of agents may be much larger (eg, dozens).

[0181] As explained above, a proxy may be dedicated to one or more applications.

[0182] The agents may vary in number and functionality depending on the embodiment. For example, an agent may be declared or installed prior to executing the method (eg, via an agent list accessible through a human-machine interface).

[0183] For example, upon receiving the message MSG 4 ={TXT="translate trout into English", ID_APP=ID_TTXT}, the semantic analysis module produces the message TRANSLATE("trout", "EN") which can be interpreted by the translation agent TRANS_AG.

[0184] The agent TRANS_AG interprets the message and returns a result RES describing the actions to be performed to process the command. 4 :

[0185]

[0186] For example, upon receiving MSG 5 ={ZONE_TYPE:WEB_URL_SEARCH="buy stock X for 20Euros", ID_APP=ID_SRCHENG, IN=oblong, PROX= magnifying glass}, the semantic analysis module generates the message STOCKS ("BUY", "X", "20") which can be interpreted by the agent STOCK_AG.

[0187] The agent STOCK_AG interprets the message and returns a result RES describing the actions to be performed to process the command. 5 :

[0188]

[0189] In another example, if the semantic analysis module analyzes the string "insert last image received by email", the semantic analysis module may generate an interpretable message to a messaging agent MSG_AG configured to retrieve a list of sent emails, content of emails and attachments, and the like.

[0190] In one or more embodiments, the knowledge server SRV_KNOW may optionally return the result provided by the agent to the client CLT_KNOW (which will then, for example, be used to replace the string generated by the user, as already explained).

[0191] In at least some embodiments, the results of the execution may consist of, for example, text, images, multimedia content, or a URL.

[0192] In at least some embodiments, the knowledge server SRV_KNOW also sends back the type of the content (text, image, video, URL, etc.) to the client CLT_KNOW.

[0193] In at least some embodiments, when the knowledge server determines that the string is not a command, it sends back a response indicating that the received string was not recognized as a command (eg, by using a keyword indicating such non-recognition, such as "NONE").

[0194] For example, by reconsidering the message MSG 1 To MSG 4 For example, the knowledge server SRV_KNWL can determine the message MSG 1 , MSG 2 , MSG 3 The string is not a command.

[0195] In at least some embodiments, the electronic device 101 may include an action management module MOD_GA for converting the response of the client CLT_KNOW into a list of actions on the window of the relevant active application (similar to user actions via an input peripheral device).

[0196] Therefore, in at least some embodiments, the action management module MOD_GA may be configured to:

[0197] i / deletes the string; and

[0198] ii / Optionally, an object is inserted in place of the string, for example at the position POS identified by the module MOD_OCR.

[0199] In at least some embodiments, to delete a character string, the action management module MOD_GA simulates for the active application the pressing of a key “DELETE” on the keyboard intended to delete the last entered character(s).

[0200] In at least some embodiments, to insert an object in place of a character string, the action management module MOD_GA copies the object to the clipboard and pastes the content of the clipboard at position POS.

[0201] So, in the above example:

[0202] - Upon receiving the response RES 4 , the action management module MOD_GA replaces the string "translate trout into English" with the string "trout" in the text area ZT;

[0203] - Upon receiving the response RSP 5 When the action management module MOD_GA replaces the string "buy stock X for 20 Euros" with the URL http: / / www.bourse.com#BUY=X#VAL=20€ in the shape of an ellipse near the magnifying glass.

[0204] Furthermore, some commands from the user may not require the insertion of new content into the active window. This is the case, for example, when the command is a control command for a home automation device, such as "turn on the light". In such an embodiment, the text corresponding to the command may simply be deleted, and the command to be executed is further sent to the system or related application (e.g., to the home automation system).

[0205] According to an embodiment, the method may be operated systematically or activated after a command is implemented (eg, via a manual action of a user of the electronic device 101 ).

[0206] Similarly, in some embodiments the method may be deactivated, for example via a manual action of a user.In this embodiment, the device S may comprise a module MOD_ACT for activating and deactivating the method.

[0207] Figure 5 The steps of executing the method in at least some embodiments are shown. In this example, the method is implemented by an application executed by the electronic device 101 in the background.

[0208] It will be assumed that during a first step ACT, the method has been activated by a user and that the method comprises a subscription to the operating system OS during a step ABON to be informed of changes of active applications and user inputs.

[0209] In the example shown, during step OBT, the method may obtain at least first data from an input device / peripheral device. The data is generated, for example, from an interpretation of an action performed by a user of the electronic device 101 at the level of the input peripheral device. The action is, for example, a series of key presses detected at the level of the keyboard (103b) of the electronic device 101. In this example, the data may correspond to at least one character and / or string representing an interpretation of the key presses performed by the user.

[0210] In the example shown, at least one data may include, for example, a fixed number n (n integers greater than 1) of characters from an input peripheral device, the obtained (one or more) characters corresponding to the last n characters typed / entered by the user at the keyboard level. Specifically, the (one or more) characters just typed / entered by the user are added / stored in one or more buffers (e.g., with a fixed maximum size), such as a FIFO (first in, first out) type list. The added characters fill the list until the list is full. Once the list is full, each character added will replace the oldest character contained in the list. Such an embodiment makes it possible to have the last (one or more) characters (e.g., the last (one or more) words) entered by the user at the input peripheral device level of the electronic device 101 in the memory at any time, regardless of the input area of ​​these characters.

[0211] As mentioned above, either a single buffer is provided for all characters input independently of the active application, or a buffer is used for each active application or for each input area of ​​each active application.

[0212] exist Figure 5 In the example of , during step IMG, the method comprises obtaining an image representing content displayed by a display peripheral device 105 of an electronic device 101. The image is obtained, for example, from software executed by the electronic device 101, which is capable of capturing the graphical content of the display peripheral device 105 of the electronic device 101 (e.g., a software probe).

[0213] According to another example, the image may be obtained from a third party device (eg, a camera or smartphone) positioned to capture content displayed by the display peripheral 105 of the device 101. The latter case may involve transmission of the image to the device 101 by the third party device.

[0214] exist Figure 5In the example of , optical character recognition (step OCR) can, for example, be applied to the image obtained. Optical character recognition makes it possible to obtain at least a partial transcription of the text contained in the image obtained / generated in the form of a string / sequence, and also to associate its position within the image obtained with at least one transcribed character (for example with each transcribed character).

[0215] According to at least one embodiment, optical character recognition can be performed by third-party software and the result, i.e., a transcription of the text (string / character sequence and optionally their associated positions), is transmitted by the third-party software to an application implementing the method that is the object of the present application.

[0216] Once the transcription is obtained, the method may include searching (step RECH) (and if the search is positive, detecting) characters within a sequence of transcribed characters stored in at least one buffer (step OBT). When the search is positive, the method determines (step DET) whether the detected string of characters constitutes a user command. The determination is made based on the string of characters and the textual context of the application that was active when the string of characters was generated. The context may include:

[0217] - The identifier of the active application;

[0218] - information about graphical objects detected by the module in the vicinity of the string;

[0219] -The date and / or location at which the electronic device was used when the string was obtained.

[0220] For example, considering the location where the electronic device is used allows determining the target (e.g., in terms of a target device or application) of a home automation command. For example, depending on the location where the device is used, the command "turn on the lights" may be interpreted as turning on the lights in a home or office, and thus may be addressed to different home automation applications or devices depending on the location where the electronic device is used.

[0221] For example, taking into account the date of use of an electronic device makes it possible to determine the action to be implemented to process a home automation command. For example, depending on the age of use of the device, the command "put on standby" can be interpreted as putting the device in standby (or even switching the device off) with different standby levels (particularly in terms of energy saving and recovery time).

[0222] In some embodiments, the method may also be used to interpret administrative commands for business tools. For example, for an employee entering time spent on some tasks in a time tracking tool, the method may allow them to enter a command of type "value 3H" in the context of an active application, which is interpreted as feeding the time tracking tool with the duration and an identifier of the active application.

[0223] In some embodiments, when the method detects that the string corresponds to a command:

[0224] - it deletes the string (step SUPPR);

[0225] - its execution command (step EXEC); and

[0226] - It is possible to insert content representing the result of the execution of the command at the position POS of the deleted text (step INSER).

[0227] In the above examples, the client CLT_KNOW and the knowledge server SRV_KNOW may be located on different entities, or one or both of them may be incorporated into the electronic device 101 .

[0228] The client-server architecture of the detailed embodiments is optional, and in some embodiments the functionality of the client and server may be implemented differently, such as within the same software function.

Claims

1. A method for executing a command generated via at least one input peripheral device (103a, 103b) of an electronic device (101) within the framework of an active application, the method being implemented by another application running in the background and comprising: - obtaining (OBT) a string of at least one character displayed on a display peripheral (105) of said electronic device (101) in a graphical environment of an active application and generated by at least one data from said at least one input peripheral (103a, 103b); - determining (DET) whether the character string constitutes a command from a user of the electronic device (101), the determination being based on the character string and also based on: the activity application, and / or the date on which the string was obtained; and / or The location where the string was obtained; and when said determination is affirmative: - deleting (SUPPR) said string on said display peripheral device (105); and -Execute (EXEC) the command.

2. The method according to claim 1, characterized in that The method includes continuously storing characters input by a user in a buffer of a memory reserved for the active application.

3. The method according to claim 1 or 2, characterized in that: The determination takes into account the type of input area used for the character string.

4. The method according to any one of claims 1 to 3, characterized in that The method comprises: - determining (DPOS) the display position of the string on the display peripheral device (105); and - displaying (AFFICH) a result of the execution of said command on said display peripheral device (105) and near said position (POS).

5. The method according to claim 4, characterized in that The position (POS) is determined by optical character recognition within an image representing content displayed by the display peripheral (105).

6. The method according to any one of claims 1 to 5, characterized in that The method comprises detecting (RECH) the character string in at least a second character string (TXT2), the second character string being obtained by optical character recognition (OCR) applied to all or part of an image representing content displayed by the display peripheral.

7. The method according to claim 6, characterized in that The optical character recognition (OCR) is applied to a portion of the image corresponding to a graphical environment of the active application displayed by the display device.

8. The method according to any one of the preceding claims, characterized in that The method comprises pre-activation (ACTIV) via an input interface of the device.

9. The method according to any one of the preceding claims, characterized in that The method is executed in the following cases: - upon detection of at least the first character; and / or - upon detection of the end of a word; and / or - upon detecting a pause in the generation of said character.

10. The method according to any one of the preceding claims, characterized in that The method is performed at regular intervals.

11. The method according to any one of the preceding claims, characterized in that The determining (DET) comprises a semantic analysis of the string, and the executing (EXEC) comprises processing, by at least one agent, a result of the semantic analysis.

12. A device (S) for executing commands generated via at least one input peripheral (103a, 103b) of an electronic device (101) within the framework of an active application, characterized in that The device(s) comprises: - a module (MOD_OBT) for obtaining a string of at least one character displayed on a display peripheral (105) of said electronic device in a graphical environment of an active application and generated by at least one data coming from said at least one input peripheral (103a, 103b); a determination module (MOD_DET) configured to determine whether the string constitutes a command from a user of the electronic device (101), the determination being based on the string and also based on: the activity application; and / or the date on which the string was obtained; and / or The location where the string was obtained; - a module (MOD_SUPPR) for deleting said string on said display peripheral device (105) of said electronic device (101); and - A module for executing said command (MOD_EXEC).

Images