Electronic device with voice control

Abstract

The invention relates to an electronic device with voice control. A system for operating a program on a primary electronic device is provided that may include the primary electronic device. The primary electronic device may have a memory to store executable instructions and one or more processors to obtain contextual data related to a user of the primary electronic device when implementing the executable instructions. The one or more processors may also be configured to determine a first location on a display of the primary electronic device based on the context data, determine a voice command related to the first location on the display, and actuate a control point at the first location based on the voice command.

Description

Technical Field

[0001] Implementations herein relate generally to voice-controlled electronic devices. Background Art

[0002] Electronic devices, such as laptop computers, mobile phones, personal digital assistants (PDAs), iPads, other computing devices, etc., have become part of the daily lives of many individuals. Such electronic devices are constantly being improved to make the user's experience as pleasant as possible.

[0003] In the past decade, voice interaction between humans and electronic devices has gained a lot of traction, and voice has become the standard modality for low-throughput modality devices such as smartphones. However, the voice ecosystem is highly fragmented. Users need to use external applications such as The like talk to their electronic devices, where different applications on the electronic devices have different dictation controls. In addition, customized voice systems have been used to control only a subset of the functions of the electronic device, rather than for all functions on the electronic device.

[0004] Additionally, there is currently no reliable way to detect when someone intends to provide voice commands, speak during a conference call, etc. In a conference call setting, many times, users attempt to speak with their microphones turned off, and users often speak on an open microphone even if they do not intend to do so. Additionally, for silence detection, applications use the sound from the microphone to detect if someone is speaking. Current solutions can sometimes filter out other noise from the background or other voices that are too far from the microphone, and display a message to the user indicating that the microphone is disabled. Computer vision (CV) can also be added to detect whether a user is speaking. However, it is possible that someone is speaking to someone nearby. When this happens, CV incorrectly detects that the user intends to speak in the meeting.

[0005] Artificial intelligence (AI) is becoming a common place for use in association with electronic devices. Whether it is helping individuals make choices when shopping, customizing usage, or simply recognizing different individuals, AI is becoming more and more prominent in everyday environments. AI applications include AI algorithms that attempt to make decisions about choices to be made using numerous variables based on the information received. The AI ​​algorithm uses initial assumptions to determine the variables, and as the individual makes choices, the variables are modified to reflect the individual's choices.

[0006] There is a need to improve the control and operation of electronic devices through the use of voice commands. Summary of the invention

[0007] According to an embodiment of the present invention, a system for operating a program on a primary electronic device is provided, which may include a primary electronic device. The primary electronic device may have a memory for storing executable instructions and one or more processors for obtaining context data related to a user of the primary electronic device when implementing the executable instructions. The one or more processors may also be configured to determine a first position on a display of the primary electronic device based on the context data, determine a voice command related to the first position on the display, and actuate a control point at the first position based on the voice command.

[0008] Optionally, to determine the voice command, the one or more processors may also be configured to: detect sounds in the user's environment, determine that the user created the sound, identify at least one word from the sound, convert at least one word into speech-to-text data, and compare at least one word to a list of words associated with the first position to determine the voice command. In one aspect, the one or more processors may also be configured to determine a second position on the display in response to actuating a control point at the first position. In another aspect, the one or more processors may also be configured to determine whether data within the memory defines a control point, and dynamically adjust the control point in the memory in response to determining that the control point is in the memory. In one example, the main electronic device may include at least one sensor that communicates with the one or more processors, and the one or more processors obtain contextual data from the at least one sensor.

[0009] Optionally, the one or more processors may utilize computer vision to determine the first position on the display. In one aspect, the one or more processors may also be configured to obtain contextual data from communications from a secondary electronic device. In another aspect, the one or more processors may also be configured to determine that the control point is within the first position before actuating the control point. In one example, determining the voice command associated with the first position may include analyzing the contextual data using an artificial intelligence application.

[0010] According to an embodiment of the present invention, a method is provided, wherein under the control of one or more processors configured with executable instructions, the method may include obtaining context data related to a user of a primary electronic device. The method may also include determining a first location on a display of the primary electronic device based on the context data, determining a voice command related to the first location on the display, and actuating a control point at the first location based on the voice command.

[0011] Optionally, determining the voice command may include: detecting sounds in the user's environment, determining that the user created the sound, identifying at least one word from the sound, converting the at least one word to speech-to-text data, and comparing the at least one word to a list of words associated with the first position to determine the voice command. In one aspect, the method may also include determining a second position on the display in response to actuating a control point at the first position. In another aspect, the method may include determining whether data in a memory defines a control point, and dynamically adjusting the control point in the memory in response to determining that the control point is in the memory. In one example, the method may also include obtaining contextual data from a communication from a secondary electronic device. In another example, the method may include determining that the control point is within the first position before actuating the control point. In yet another example, determining the voice command associated with the first position may include analyzing the contextual data using an artificial intelligence application.

[0012] According to an embodiment of the present invention, a computer program product is provided, which may include a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including a computer executable code for obtaining context data related to a user of a primary electronic device. The computer program product may also determine a first position on a display of the primary electronic device based on the context data, determine a voice command related to the first position on the display, and actuate a control point at the first position based on the voice command.

[0013] Optionally, the computer program product may also detect sounds in the user's environment, determine that the user created the sound, recognize at least one word from the sound, convert the at least one word to speech-to-text data, and compare the at least one word to a list of words associated with the first position to determine the voice command. In one aspect, the computer program product may also determine a second position on the display in response to actuating a control point at the first position. In another aspect, the computer program product may also determine whether data in the memory defines a control point, and dynamically adjust the control point in the memory in response to determining that the control point is in the memory.

[0014] According to an embodiment of the present invention, a system for operating a program on a primary electronic device is provided, which may include a primary electronic device having a memory for storing executable instructions and one or more processors. When implementing the executable instructions, the one or more processors are configured to: obtain context data related to a user of the primary electronic device, determine whether the program is muting the user based on the context data, determine whether the user intends to communicate using voice via the program based on the context data in response to determining that the program is muting the user, and when it is determined that the user intends to communicate using voice, automatically activate the program to unmute to allow the user's voice to be transmitted to the program.

[0015] Optionally, the program can be a teleconferencing application. In one aspect, in order to determine that the user intends to communicate using sound, one or more processors can be configured to: analyze context data to determine that the user is in front of the main electronic device, and use computer vision (CV) to determine that the user's gaze is at the main electronic device. In another aspect, the system can also include a sensor coupled to one or more processors and configured to obtain context data. In one example, the sensor can be at least one of a camera, a microphone, an infrared sensor, or a temperature sensor. In another example, one or more processors can also be configured to determine the user's workspace and determine whether the user is in the workspace in front of the main electronic device. In yet another example, one or more processors can also be configured to identify another individual or animal in the workspace, and filter another individual or animal from the image data obtained by one or more processors. In one embodiment, one or more processors can also be configured to determine that the user's voice is a voice command and implement a voice command. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic block diagram of a communication system according to embodiments of the present invention is shown.

[0017] Figure 2 A schematic block diagram of an electronic device according to embodiments of the present invention is shown.

[0018] Figure 3 A plan view of a display of a primary electronic device according to embodiments of the present invention is shown.

[0019] Figure 4 A plan view of a display of a primary electronic device according to embodiments of the present invention is shown.

[0020] Figure 5 A plan view of a display of a primary electronic device according to embodiments of the present invention is shown.

[0021] Fig. 6A A plan view of a display of a primary electronic device according to embodiments of the present invention is shown.

[0022] Figure 6B A plan view of a display of a primary electronic device according to embodiments of the present invention is shown.

[0023] Figure 7 A plan view of a display of a primary electronic device according to embodiments of the present invention is shown.

[0024] Figure 8A schematic flowchart of a method for controlling an application on a primary electronic device according to an embodiment of the present invention is shown.

[0025] Fig. 9 A schematic flow chart of a method for controlling an application on a primary electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] It will be readily understood that, in addition to the described example embodiments, the components of the embodiments as generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of the example embodiments as illustrated in the drawings is not intended to limit the scope of the claimed embodiments, but merely represents example embodiments.

[0027] References throughout this specification to "one embodiment" or "an embodiment" (etc.) mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment," "in an embodiment," etc., appearing in various places throughout this specification are not necessarily all referring to the same embodiment.

[0028] In addition, in one or more embodiments, the described features, structures or characteristics may be combined in any suitable manner. In the following description, many specific details are provided to provide a thorough understanding of the various embodiments. However, those skilled in the relevant art will recognize that the various embodiments may be practiced without one or more specific details, or practiced in the case of other methods, parts, materials, etc. In other cases, well-known structures, materials or operations are not shown or described in detail to avoid confusion. The following description is intended only by way of example, and some example embodiments are simply shown.

[0029] The term "context" shall mean any and all parameters, characteristics, variables, attributes, and the like that may be used to make determinations related to an environment, workspace, user, voice command, and the like. Context may be used to make a determination, or as part of a calculation, formula, decision tree, and the like. Context data may be obtained from sensors of a primary electronic device, sensors of a secondary electronic device, storage devices of a primary electronic device or a secondary electronic device, determinations made based on information transmitted from a secondary electronic device to a primary electronic device, determinations made based on data detected by a primary electronic device or a secondary electronic device, data detected by a primary electronic device or a secondary electronic device, and the like. Context data may include user gaze, eye movement, position of a user or a user's head relative to a display screen, identification of a person, presence of another person in an environment or workspace, motion data, location data, personal data, and the like.

[0030] The terms "electronic device context model" and "EDC model" shall mean a high-level model and / or algorithm, including a machine learning model or algorithm, an artificial intelligence model or algorithm, etc., which utilizes context data to identify contexts related to users of electronic devices. The context data may be received from a primary electronic device, a secondary electronic device, etc., including from sensors utilizing image recognition software, gesture recognition software, voice recognition software, global positioning system (GPS) software, etc. The EDC model determines the context of an individual's environment or workspace based on analysis of context data obtained from a primary electronic device and / or context data obtained from another electronic device (e.g., a secondary electronic device, an environmental electronic device, etc.). For example, based on head position, eye movement, etc. (e.g., context data), the EDC model may determine where a user is looking at a display screen. Such context may be used by a voice control application to determine a user's voice command. As an example, if the EDC model determines from context data that a user is looking at a context of a save control point, such that when the user says the word "save", the voice control application may use the context to cause the program to perform a save function.

[0031] As used herein, the term "control point" refers to any and all buttons, symbols, indicia, menus, etc., presented on a display screen by an application or program that can be actuated to perform a function in the application or program. For example, a program may be Word TM A document may include control points "File", "Home", "Insert", "Draw", "View", etc. at the top of the display screen. When one of these control points is actuated, a drop-down menu may appear that may include additional control points. To this end, the function of these control points is to present additional control points. Additional control points, such as "Save", "Print", "Share", etc. may be additionally actuated to save data on the display to the device's memory, send a command to a printer to print the content on the display screen, send a document, presentation, etc. on the display screen to another in the form of an email, etc. In other examples, the control point may be a symbol, such as a disk that may be present on the screen, which automatically saves data to the memory of the electronic device when actuated.

[0032] As used herein, the term "focused content" refers to the content that a user of a primary electronic device is viewing on a display. For example, a user may view a first location, such as the upper left corner of a screen that includes a "Home" name. The voice control application may utilize contextual data such as eye gaze, iris position, head angle, etc., and use an EDC model to determine the focused content (e.g., the "Home" name) on the display screen of the primary electronic device.

[0033] As used herein, the term "voice command" refers to any sound that can be detected by a voice command application that causes a program or application presented on the display of a primary electronic device to perform a function. The sound can be any and all sounds that occur. The command can be a single word, phrase, etc. that will cause the function of the primary electronic device. For example, a command can be a single term, such as "bold", "save", "print", etc. Alternatively, the command can be a phrase or sentence, such as "bold the entire paragraph". Similarly, a phrase can be "spell check document", "spell check previous word", "insert footnote", "find word main", "new slide", etc. In an example embodiment, the voice command causes the actuation of a control point on the display of the primary electronic device.

[0034] The term "primary electronic device" shall mean any device, system, controller, etc. that can monitor and transmit data and information related to a user. The primary electronic device may include a smart phone, smart watch, smart remote control, smart clothing, vehicle controller, etc. that can obtain contextual data. The primary electronic device is also configured to communicate with secondary electronic devices to receive contextual data and information related to an individual, an environment of interest, a workspace, etc., and can be used within an EDC model. The primary electronic device may communicate with one or more secondary electronic devices via one or more wireless protocols including Bluetooth, GSM, infrared wireless LAN, HIPERLAN, 4G, 5G, satellite, etc., via wire (i.e., USB) or via air.

[0035] The term "secondary electronic device" shall mean any device, system, controller, etc. that can monitor and transmit data and information related to an individual, environment of interest, workspace, etc. that is not a primary electronic device. The primary electronic device utilizes the EDC model to make determinations related to contextual data obtained by the secondary electronic device. Secondary electronic devices include Internet of Things (IoT) devices, smart phones, smart watches, smart TVs, tablet devices, personal digital assistants (PDAs), voice-controlled smart personal assistant service devices - including Google Smart speakers, etc. As an example, if a user is watching a smart TV (e.g., a secondary electronic device) while utilizing a primary electronic device, the smart TV may provide information such as program descriptions, program content, etc., which may be used by the EDC model to determine the content of any dictation provided by the user. Alternatively, other secondary electronic devices may indicate that other individuals are in the room, such that the EDC model may determine that the user is talking to people in the environment and is not providing voice commands.

[0036] The term "workspace" as used herein refers to an area defined by the union of all monitors connected to an electronic device. As an example, if a user is looking at any connected monitor, the user is considered to have noticed the communication with the electronic device. In an example, a sensor such as a camera device can be attached to the electronic device and use eye gaze and head posture information calculated using computer vision methods to identify where the user is looking to determine whether the user is looking at the workspace defined by the union of external monitors.

[0037] As used herein, the terms "neural network" and "machine learning" refer to artificial intelligence algorithms that learn from various automatic or manual feedback such as observations and / or data. The artificial intelligence algorithm is adjusted in multiple iterations based on the observations and / or data. For example, the artificial intelligence algorithm is adjusted through supervised learning, unsupervised learning, and / or reinforcement learning (e.g., customer feedback). Non-limiting examples of artificial intelligence algorithms are decision trees, K-means, deep learning, artificial neural networks, etc.

[0038] As used herein, the phrases "dynamically adjust" or "dynamically adjusting" means to change or vary in real time, in response to conditions or otherwise.

[0039] As used herein, the phrase "real time" shall mean a time that occurs simultaneously or substantially simultaneously with the occurrence of another event or action. For the avoidance of doubt, as an example, a dynamically adjusted object or device changes instantly or within a second or two.

[0040] The term "network resource" refers to any device, system, controller, etc. that can monitor and transmit data and information related to an individual. Network resources may include servers, applications, remote processors, clouds, etc. Network resources may communicate with electronic devices through wires via one or more wireless protocols including Bluetooth, GSM, infrared wireless LAN, HIPERLAN, 4G, 5G, satellite, etc.

[0041] The terms "obtains" and "obtaining" as used in conjunction with data, signals, information, etc. include at least one of the following operations: i) accessing the memory of an external device or remote server storing data, signals, information, etc., ii) receiving data, signals, information, etc. through a wireless communication link between the base device and the auxiliary device, and / or iii) receiving data, signals, information, etc. at a remote server through a network connection. When viewed from the perspective of the base device, the obtaining operation may include: sensing new signals in real time, and / or accessing memory to read stored data, signals, information, etc. from memory within the base device. When viewed from the perspective of the auxiliary device, the obtaining operation includes receiving data, signals, information, etc. at a transceiver of the auxiliary device, wherein the data, signals, information, etc. are sent from the base device and / or the remote server. The obtaining operation may be from the perspective of the remote server, such as when receiving data, signals, information, etc. from a local external device and / or directly from the base device at a network interface. The remote server may also obtain data, signals, information, etc. from local memory and / or from other memory such as within a cloud storage environment and / or from the memory of a personal computer.

[0042] It should be clearly understood that the various arrangements and processes, and / or one or more individual components or elements of such arrangements and / or one or more associated processing operations of such processes, described and illustrated with respect to the accompanying drawings, may be used independently of or in conjunction with one or more other components, elements, and / or processing operations described and illustrated herein. Thus, while various arrangements and processes are broadly contemplated, described, and illustrated herein, it should be understood that they are provided in an illustrative and non-limiting manner only, and furthermore may be considered merely as examples of possible operating environments in which one or more arrangements or processes may function or operate.

[0043] Systems and methods are provided for allowing a user of a primary electronic device to control and operate a program or application using voice commands. The system obtains contextual data from a primary electronic device and a secondary electronic device to determine a location on a display that a user is viewing. Such contextual data can be obtained using computer vision (CV) to determine the location where the user's eyes are looking at or focusing. From this, the focus content within the location on the display can be determined. When this occurs, the system detects sounds made by the user to determine a voice command that can be associated with the focus content. When a voice command such as "save" associated with the focus content (e.g., a symbol of a disk at a location on the display) is provided, a control point such as a disk symbol can be actuated to operate a program.

[0044] Figure 1is a block diagram of a system 100 for operating a primary electronic device 102 using voice commands according to an embodiment of the present invention. The primary electronic device 102 utilizes contextual data related to a user, the user's environment, the user's workspace, etc. to identify controllable fields or control points of a program or application and interactively associates those control points with voice commands. Control points may include tags, symbols, words, etc. that, when actuated, cause an action such as saving, printing, opening a document, closing a program, inserting a footnote, inserting a page break, creating a new slide, etc. A program may be any set of instructions for performing a function on the primary electronic device. In an example embodiment, the program includes Word 106. TM , PowerPoint TM , PDF TM 、FireFox TM , Webex TM , Google docs TM , game applications, etc.

[0045] In one example, a primary electronic device 102 is provided that can obtain contextual data related to a user of the electronic device and to the primary electronic device. The contextual data can be obtained from one or more sensors 104 of the primary electronic device 102, input from the user received by the primary electronic device, contextual data obtained by a secondary electronic device 106 and transmitted to the primary electronic device, and the like.

[0046] In one example, sensor 104 can be a camera of a main electronic device that identifies the area or position that the user is focusing (i.e., looking at) on the screen of the main electronic device 102. In other examples, sensor 104 can include beamforming microphones, passive infrared sensors, time of flight or LiDAR sensors, high-resolution red, green, and blue (RGB) cameras, high-resolution RGB wide-angle cameras, light level sensors, global navigation systems, etc. In yet another example, the sensor can be a microphone that detects sounds in the environment. In this way, the voice control application can determine whether the sound is a voice command provided by the user. Although only two sensors 104 are shown, in other examples, three or more sensors are provided. Regardless of the number of sensors, inputs from multiple sensors 104 - sensors including but not limited to cameras, microphones, infrared, and ultrasound - can be used to detect when a user appears in front of the computer, when a user speaks (using voice, CV, etc.), and when the user intends to speak to the PC.

[0047] In particular, input from sensor 104 can be used with context application 107 to identify whether a user is present in front of main electronic device 102. In one example, context application can include processing for ultrasound, infrared, CV, etc. Alternatively, context application 107 can be provided, which identifies whether there is sound from sensor 104 such as microphone. Context data application can also be used to filter sound input and determine sound characteristics. Sound characteristics can include the distance of the sound source from the sensor or microphone (e.g., whether the sound is far away from or close to the microphone), whether the detected sound is human voice, whether the detected sound belongs to the user (e.g., the context application includes a speaker ID function), etc. In one example, in order to provide human voice confirmation, context application 107 can determine whether a person is present at main electronic device 102 and is speaking. In one example, CV can be used to detect the user's mouth movement associated with the sound detected by the sensor.

[0048] In addition, the context application 107 can determine whether the user is looking at the main electronic device to verify that the user intends to speak to the computer. Likewise, in one example, CV can be used to make such a determination. Such a determination that the user expects to speak to the computer can be used to determine the voice command provided by the user, determine when the user expects to speak during a conference call, etc. To this end, once it is determined that the determined condition threshold is met, the context application can determine whether the user is looking at one or more displays, workspaces, a portion of a display, an open web page, etc., to verify that the user is providing a voice command, expects to be heard during a conference call, etc. In one example, the context application can use CV again to make such a determination. In another example, the determined conditions can include the user's presence in front of the display, the user viewing a display screen including a window with a conference call, the user's typical gestures for asking questions or making statements, attention or eye focus on a determined area of ​​the display, etc. To this end, the threshold can be two conditions, three conditions, etc. Once the conditions of this number are reached, additional analysis can be performed. If the conditions of this number are not met, the context application does not perform any additional determination.

[0049] Once the context application determines that a threshold number of determination conditions have occurred, the context application can confirm that the user's intention is to speak to the computer to provide voice commands, communicate with others on a conference call, etc. In one example, the context application can verify the user's intention by reading window information on one or more displays. Specifically, the context application 107 can determine the subject contained in the window on the screen, the size of the window or context information, the number of different displays the user is using and the windows open on each display, etc. In addition, the context application 107 can recognize the user's gesture or gaze to identify the screen and the location on the screen that the user is viewing. In one example, such a determination is made using CV.

[0050] In one example, if a user is on a conference call and has a PowerPoint presentation open on a first screen and a conference call open on a second screen, and the user is viewing or watching the presentation and asking questions while muted, the contextual application can determine that the user expects to be heard and unmute the user. Specifically, the contextual application can associate the information within the presentation with the title, subject, shared document, etc. related to the conference call, while also determining that the user is paying attention to and reading the presentation. Based on these determined conditions, the contextual application can determine that the user is asking questions about the presentation to other individuals on the call. Other conditions that can be presented include silence during the meeting (silence indicating an appropriate time to ask questions), the user using words such as "Excuse me", "Wait a minute", "Quick question", etc. indicating that the user is trying to interrupt the person who is speaking, etc. Based on the conditions so determined, the contextual application can determine that the user is asking questions and forgets to unmute their microphone, so that the contextual application automatically turns on their microphone, allowing them to ask their questions in a timely manner.

[0051] In another example, the context application 107 can define the user's workspace and use the defined workspace to provide a determined condition. For example, the workspace can be defined by the field of view of any one of the three cameras on three separate displays. Alternatively, only a portion or space within a determined distance of the camera can be a defined area. To this end, the defined area can be any space within fifteen inches of the camera of the display. In another example, the content application 107 can define a workspace determined as a physical area defined by all external monitors. The content application can receive information about the location of the external monitor from the operating system and define the workspace. In addition, the content application can use the camera of one of the devices to determine whether the user is viewing the workspace using computer vision technology. In such an embodiment, when the context application detects that the user is viewing the workspace, it determines that the user notices the context application. In one example, the context application 107 can be configured to unmute the microphone during a conference call only when the user is in a defined area and is viewing a display showing a conference call. Specifically, the context application 107 determines that the user has the intention to speak only when the user is in a defined area and is viewing the screen. Otherwise, if the user is not in the defined area, or is looking in a direction other than towards the display, and potentially talking to someone outside the camera, the contextual application 107 will not unmute the microphone.

[0052] In addition, by utilizing a defined area, the sounds and actions of other people detected outside the area will not be mistaken for voice commands or statements that the user does not expect to be heard during the call. For example, if the user is working from home and is on a conference call when the user's teenager walks into the user's office and ignores that a conference call is ongoing and asks their parents questions during such a call, the context application 107 can determine that the teenager's noise and walking actions did not occur in the defined workspace, so that the user is not unmuted to allow everyone in the meeting to hear their teenager's questions. To this end, in one example, the context application 107 may include a filter to eliminate any individuals, animals, etc. that enter the field of view of the camera of the main electronic device from being displayed. In the example, the person closest to the camera is considered to be the user, and any other presence that accidentally enters the field of view of the camera is not shown, such as teenagers, pets, other relatives, etc. This prevents embarrassing interruptions for people working at home.

[0053] Each of the sensors 104 can provide different types of information that can be utilized by one or more processors, voice control applications 108, AI applications 110, network AI applications, etc. The voice control application 108 can obtain contextual data from the environment. This includes contextual data obtained from manual input, sensors 104, secondary electronic devices 106, etc. In one example, a sensor that is a camera detects the user's eyes to determine where the user is looking on one or more screens.

[0054] In another example, the primary electronic device 102 may be coupled to and communicate with a secondary electronic device 106 that obtains contextual data. The contextual data for the secondary electronic device 106 may be obtained in the same manner as the primary electronic device 102, including via sensors. For example, the secondary electronic device may be a smart TV, tablet, or the like that includes a video feed of a presentation, lecture, meeting, or the like that is being watched or viewed by a user. To this end, the user may choose to dictate notes related to the video feed, presentation, lecture, meeting, or the like on their own personal electronic device that is the primary electronic device 102. The secondary electronic device 106 may then transmit contextual data related to the video feed, presentation, lecture, meeting, or the like that may be utilized by the primary electronic device to not only determine the user's intent (e.g., to dictate notes related to the video feed, presentation, lecture, meeting, or the like), but also to help determine the words being spoken by the user that will be used as notes.

[0055] By using contextual data, the voice control application 108 can make a determination related to the user's intent or desire to actuate a control point. For example, if the camera determines that the user is looking at a first location on the screen, which is the upper portion of the screen with control points or functions such as "Open", "Save", "Save As", "Underline", "Font Size", "Quote", etc., then a determination can be made by the voice control application 108 based on the voice command that the user is attempting to command the program to provide one of these functions. However, alternatively, if the user's gaze is looking at a second location on the screen, such as the middle of the screen with an open program, application, document, etc., it can be determined that the user is attempting to dictate text into the program, application, document, etc. In particular, the voice control application may include an EDC model that can utilize contextual data obtained from the environment, the user, etc. to determine the voice command.

[0056] In one example, the voice control application 108 may include an artificial intelligence (AI) application 110 for making determinations related to the contextual data and the user's voice command. In one example, the AI ​​application is used with the EDC model to change the variables of the EDC model to facilitate determinations related to the voice command.

[0057] The voice control application 108 and the AI ​​application 110 can make a determination related to the voice command to operate the program or application of the main electronic device 102. The voice command can include any and all words, statements, sounds, etc. made by the user that can cause the actuation of the control point on the display of the electronic device. The voice command can include a user's command or request such as "save the presentation", and a speech, statement, sound, etc. made by the user to be dictated or placed in a document, presentation, etc. The voice control application 108 may include natural language understanding (NLU), automatic speech recognition (ASR), text to speech synthesis (TTS), algorithms, models, functions, etc., to interpret natural language input in spoken form to infer intent, and perform actions based on the inferred user intent. In one example, NLU, ASR, TTS, etc. can be provided by the EDC model. In another example, the voice control application 108 can be sent to a cloud natural language processing (NLP) service to analyze commands, comments, etc. In one example, the voice control application 108 can operate in conjunction with the context application 107 to verify that the voice command matches the topic related to the context data obtained by the context application 107. The voice control application 108 can determine dialects, accents, different languages, etc. For example, the primary electronic device 102 can receive a user request in the form of a natural language command, request, statement, narrative, and / or query. The user request can seek a task through the primary electronic device. Therefore, the primary electronic device 108 can perform the requested task. The user request can be in the form of natural language, request, statement, narrative, and / or query. The user can also submit a user request for performing a task, for example, by stating "make the text on the screen bold."

[0058] In one example, the foreground application can communicate with the voice control application 108 of the main electronic device 102 to act as a focus application to determine the focus content presented on one or more screens or displays. The focus content can be a location, application, command bar, etc. located at the location where the user is watching or focusing. In the example, gaze can be used to identify which location or application on one or more screens the user is viewing. In an example implementation, the application can be a document, presentation, website, etc. that includes user-available functions. The camera of the main electronic device 102 can recognize the user's iris position and head gesture. Then, based on the layout of the screen or display, the position of the camera relative to the available display, and the position of the focus application relative to the display, the voice control application 108 can determine the location or application that the user is viewing, so that the focus content can be determined. In addition, the user can trigger various actions within the application to change the focus content. For example, the user can call a menu with various items to select from, or can bring a new sub-window that is part of the main application. In this case, the new content can become the focus content. The focus sub-content can be obtained by creating a difference between the current image of the application and comparing it with the image of the application before the sub-window is created. When the user closes the sub-content, the application becomes the focus content again. In one example, the CV may be used to analyze the focus content to determine the various control points of the application. Although in one example the control points are words, alternatively, the control points may also be symbols or graphic elements.

[0059] The voice control application 108 includes program instructions that can be accessed by one or more processors 112 to direct one or more processors 112 to implement the methods, processes and operations described herein, including program instructions associated with the EDC model. The voice control application 108 can manage the operation of one or more other components of the main electronic device 102 to parse the spoken words of the received voice command, analyze the command and execute the command. In an embodiment, the voice control application 108 is used together with a conflict resolution algorithm. For example, the conflict resolution algorithm can be used to determine which voice commands the voice control application 108 responds to, and which voice commands the voice control application 108 rejects or ignores.

[0060] In one example, the voice control application 108 operates one or more processors 112 of the main electronic device 102 to determine that the source (e.g., user) of the voice command is a specific person who speaks the voice command to perform the first function. In an embodiment, the source of the voice command is determined based on a voice recognition analysis of the voice command. For example, one or more processors analyze the audio characteristics of the voice command to associate the voice command with the voice of a specific user (e.g., person). The audio characteristics may include intonation, speech rate, sound frequency, sound amplitude, sound pattern, word pattern, etc. The one or more processors performing the voice recognition task may include at least one processor of a computer and / or server located away from the main electronic device 102 and connected to the main electronic device 102 via a network (e.g., the Internet). Optionally, the one or more processors 112 may perform at least a portion of the voice recognition task.

[0061] One or more processors 112 can utilize speech recognition algorithms enhanced with machine learning to improve use and experience. For example, the speech recognition algorithm can include a neural network. By determining the individual providing the command, the voice command application 108 can more easily interpret the command and the user's expectations. To this end, different users of the main electronic device can each have an individual profile associated with the user. Some users may have different preferences regarding the use of voice commands. For example, a user may only want to use voice commands to dictate information into a text document. Alternatively, in addition to being able to dictate text into a document, other users may want to use all voice command features, including functions that execute applications, such as saving, opening, inserting symbols, etc. By having a profile, the user experience can be customized accordingly.

[0062] In one example, the system 100 may include one or more servers 120. By way of example, the primary electronic device 102 may be a mobile device such as a cellular phone, a smart phone, a tablet computer, a personal digital assistant, a laptop / desktop computer, a gaming system, a media streaming hub device, an IoT device, or other electronic terminal including a user interface and configured to access the network 140 through a wired or wireless connection. As a non-limiting example, the primary electronic device 102 may access the network 140 through a wireless communication channel and / or through a network connection (e.g., the Internet). In one embodiment, the primary electronic device 102 communicates with the network resource 130 via the network. The network resource 130 may be a server, an application, a remote processor, a cloud, etc. In one example, the network resource 130 is one or more processors of the secondary electronic device 106 that communicate with the electronic device 102 through the network 140. The network 140 may represent one or more of a local area network (LAN), a wide area network (WAN), an intranet, or other private network that may not be accessible to the general public, or a global network such as the Internet or other publicly accessible networks. The AI ​​application 110 may utilize such a network to obtain additional contextual data or information for making determinations.

[0063] Additionally or alternatively, the main electronic device 102 may be a wired or wireless communication terminal, such as a desktop computer, a laptop computer, a network-ready television, a set-top box, etc. The main electronic device 102 may be configured to access the network using a web browser or a local application executed thereon. In some embodiments, the main electronic device 102 may have a physical size or form factor that enables it to be easily carried or transported by a user, or the main electronic device 102 may have a physical size or form factor that is larger than a mobile device.

[0064] Figure 2 According to the embodiment Figure 1 1 is a simplified block diagram of a primary electronic device 102. Although described as a primary electronic device 102, in other embodiments, the device shown may be a secondary electronic device. The primary electronic device 102 includes components such as one or more wireless transceivers 202, one or more processors 204 (e.g., microprocessors, microcomputers, application specific integrated circuits, etc.), and one or more local storage media (also referred to as memory portions) 206.

[0065] Each transceiver 202 can communicate using known wireless technologies. Exemplary operations of the wireless transceiver 202 in conjunction with other components of the primary electronic device 102 can take various forms and can include, for example, the following operations: Upon receiving a wireless signal, the components of the primary electronic device 102 detect a communication signal from the secondary electronic device 207, and the transceiver 202 demodulates the communication signal to recover the incoming information sent by the wireless signal, such as a response to a query request, voice, and / or data. The one or more processors 204 format the outgoing information and transmit the outgoing information to one or more of the wireless transceivers 202 for modulation into a communication signal. The wireless transceiver 202 transmits the modulated signal to a remote device, such as a cellular tower or a remote server (not shown).

[0066] The local storage medium 206 may encompass one or more memory devices in any of a variety of forms (e.g., read-only memory, random access memory, static random access memory, dynamic random access memory, etc.), and may be used by one or more processors 204 to store and retrieve data. The data stored by the local storage medium 206 may include, but need not be limited to, operating systems, applications, context data obtained, and information data. Each operating system includes executable code that controls the basic functions of the device, such as interaction between various components, communication with external devices via the wireless transceiver 202, and storage and retrieval of application and context data to and from the local storage medium 206. In one example, the transceiver may communicate with a secondary electronic device 207. In addition, the transceiver may also communicate with a remote device 211 having a remote database to transmit context data and determinations made by the one or more processors 202, and obtain context data from one or more secondary electronic devices 207.

[0067] In one embodiment, the electronic device 102 further includes a Figure 1) communication interface 208. Communication interface 208 may include one or more input devices 209 and one or more output devices 210. Input device 209 and output device 210 may each include various visual, audio and / or mechanical devices. For example, input device 209 may include visual input devices such as optical sensors or cameras, audio input devices such as microphones, and mechanical input devices such as keyboards, keypads, selection hard buttons and / or soft buttons, switches, touch pads, touch screens, icons on touch screens, touch-sensitive areas on touch-sensitive screens, and / or any combination thereof. Similarly, output device 210 may include visual output devices such as liquid crystal display screens, one or more status indicators that may be light elements such as light-emitting diodes, audio output devices such as speakers, alarms, and / or buzzers, and mechanical output devices such as vibration mechanisms. The display may be touch-sensitive to various types of touches and gestures. As another example, output device 210 may include a touch-sensitive screen, a non-touch-sensitive screen, a pure text display, a smart phone display, an audio output (e.g., a speaker or a headphone jack), and / or any combination thereof. In one example, the output device 210 may be used to provide suggestions or recommendations to the user. The suggestions or recommendations may be in text form, may include directional arrows indicating where to go, may be auditory commands, and the like.

[0068] As about Figure 1 As described, the electronic device 202 also includes a first sensor 212, a second sensor 214, an artificial intelligence (AI) application 218, and a voice control application 220. All of these components can be operably coupled to each other through one or more internal communication links such as an internal bus, and can communicate with each other. The first sensor 212 and the second sensor 214 are both used to obtain contextual data related to the user or the environment. The type of information can include visual, auditory, tactile, infrared, etc.

[0069] In one example, by obtaining information related to a user or an environment, the one or more processors 204 may determine a profile related to an individual to provide settings for the first sensor 212 and the second sensor 214. Specifically, the profile may be related to the individual and include operating settings of the first sensor 212 and the second sensor 214 based on conditions within the environment. To this end, on a primary electronic device shared by multiple individuals, a first individual may have a first profile and a second individual may have a second profile.

[0070] In one embodiment, the AI ​​application 218 and the voice control application 220 are stored in the storage medium 206 and each includes executable code. Both the AI ​​application 218 and the voice control application 220 obtain information, including contextual data, from the first sensor 212, the second sensor 214, and other sensors, information input by the user, remote devices, etc. For example, the AI ​​application 218 can obtain contextual data related to the user and the user's environment, including the user's gaze, the position of the control point on the display screen of the electronic device, etc., to make a determination related to the user's voice command and the focus content on one or more screens or displays related to the voice command. The AI ​​application 218 can also receive auxiliary contextual data related to electronic devices in similar locations from the remote device 211 to provide more accurate calculations related to the contextual data of the environment of interest, voice commands, and focus content.

[0071] Figures 3 to 7 Provided Figure 1 to Figure 2 Examples of uses of systems and electronic devices in. Figure 3 to Figure 4 An example screen 300 or display is shown in which feature extraction is utilized. In particular, CV can be used to analyze the focus content 302 of an application 304 to obtain various control points 306 of the application 304. In one example, the control point 306 is a word, and in other examples, the control point 306 can be a symbol, a graphical element, etc. Figure 4 A highlighted control point 306 is shown to illustrate different locations of screen 300 that may be considered control points 306, where the box may represent focus content 302. Additionally, as shown, the control point may be part of the application itself, such as command bar 310, or may be a place in application 304 where text (e.g., text area 312) is provided.

[0072] In one example, when the control point is a word, the voice command application may include optical character recognition (OCR) that analyzes the word and determines the content indicated by the text. To this end, OCR may provide the location and size of the recognized text with an area of ​​interest that is considered to be the focus content 302. The focus content is determined based on the user's gaze or eye focus and / or the voice command provided. Figure 3 to Figure 4As shown, OCR can detect the control point 306 of the application and the text that the user has entered into the application. According to the needs of the voice command, such user input text can be filtered out from the screen. In one example, the position of the recognized text is used to filter such text to select what text can be classified as the control point 306. Usually the top of the application is where the control is located, while in other applications, the control can be located on the left side of the screen or the right side of the screen. Therefore, this contextual data related to the application can help determine in real time the relationship between the text presenting the control point 306 and the text already entered by the user. Alternatively, another example of how to filter the control point 306 is to use an AI application (e.g., machine learning) to understand that the content in the application changes over time (where the text input by the individual presents changes, and the position of the command is usually constant). Therefore, the AI ​​application is able to determine over time where the exact position of the control point 306 representing the command of the application is.

[0073] In addition to text, the voice command application can also identify control points that are symbols. For example, symbols often have additional information that is provided when a user interacts with the symbol. In one example, when a mouse is actuated to hover an arrow or cursor over the top of a symbol, additional information is provided. To this end, the user can expand the list of control points 306 in the application with their relative positions. In an example, an application programming interface (API) can be utilized, or documentation can be provided to automatically extract control points 306 and their locations in real time.

[0074] In addition, if Figure 5 As shown, the control point may include a shortcut 500. A command may be associated with a shortcut 500 in an application, such as when a drop-down menu is provided. As shown, the shortcut 500 may include a function name 502 and a shortcut 504. In particular, sometimes users do not know the specific function they expect to use, so that the shortcut menu is used before providing a voice command. Alternatively, the shortcut is utilized because the system is not operated for its intended purpose of eliminating such searches and the use of shortcut menus. Therefore, in one embodiment, once the shortcut 500 is opened, the voice command application may automatically debug the system to determine why the voice command did not cause the desired action and why the user had to use the shortcut 500. Alternatively, the user may also provide the command "start debugging" to also cause debugging to occur.

[0075] Fig. 6A and Figure 6BAnother screenshot 600 of the features of the system in this article is shown. In this example embodiment, a silent action can prompt the system to begin operating based on the user's voice command. As shown, a menu 602 can be provided with many text fields 604. When the user selects the selected text field 606, the voice command application is automatically activated so that when the user begins to provide voice commands, such voice commands can be implemented. Upon selection, OCR automatically begins to make determinations related to the position on the screen that the user is viewing in real time to determine the control point. In addition, a sensor such as a microphone can begin recording sound. In particular, although in one example context data can be used to determine when to start using a microphone to record the user's voice commands, in this embodiment, the selected text field provides this functionality.

[0076] Figure 7 Yet another screenshot 700 of an application 702 including commands 704 and a text area 706 is shown. In this embodiment, eye tracking and head gesture detection CV functionality is used to provide a gaze window 708 with focus content 710 to determine the approximate area the user is looking at. Only commands provided within the gaze window 708 can then be executed via voice commands. By showing the user the gaze window, the user knows that if a voice command is desired, the user must change their gaze to change the gaze window 708. By providing voice commands based only on commands within the gaze window, accuracy of the commands being executed is improved, thereby reducing false triggers.

[0077] In conclusion, Figures 1 to 7 Many systems and examples of how to identify control points on the screen of a primary electronic device are shown. Such control points can be identified via contextual data, including using CV to determine the user's gaze and head position to determine the position on the screen that the user is viewing. Once these control points are determined, they can be saved as data in a memory or in a cloud environment to define the control points. The defined control points can then be used to dynamically adjust the control points associated with the application. In this way, the user can then use their gaze and head position to signal that the user wishes to provide a voice command associated with the control point, so that the action can be automatically performed by one or more processors in response to the voice command. Because voice commands can be three times more efficient than typing, efficiency is improved.

[0078] Figure 8 A block diagram of a method 800 for controlling the operation of an application of an electronic device is shown. The method 800 may be implemented using Figures 1 to 7 In particular, the method can utilize context data and context applications to determine whether a voice command is being provided and implement the voice command.

[0079] At 802, one or more processors of the primary electronic device obtain contextual data about an environment of interest. One or more processors may include a context application that obtains contextual data from a first sensor, inputs information into an electronic device, obtains contextual data from a secondary electronic device, and the like. Contextual data may include data obtained using methods such as hearing, vision, touch, infrared, temperature, etc. The environment of interest may be a room, a residence, a home, an office building, a vehicle, an auditorium, and the like. In one example, the contextual data includes eye gaze data and head position data obtained from a CV method as described herein. In other examples, the sensor may detect movement of the user's mouth or other actions that may be used to determine an expected voice command.

[0080] At 804, one or more processors of the primary electronic device determine whether to utilize program control using voice commands. In one example, the determination is made based on context data utilizing a context application. In another example, the determination is made based on a manual input or selection by a user. In yet another example, a user profile is determined, and based on the user profile, a voice control function is activated.

[0081] At 806, if it is determined that voice control is used to control the program, the one or more processors activate the microphone to begin detecting sounds in the environment to determine whether a voice command is provided. In one example, the one or more processors determine the source of each sound, including whether the sound is from the user or from another source in the environment. In an example, voice recognition software and / or hardware can be used to determine that the sound is from the user.

[0082] At 808, one or more processors obtain voice data from the user via sensors such as microphones, secondary electronic devices, etc., and generate voice-to-text (VTT) data. Once the user is identified as the source of the sound, the voice control application analyzes the sound to determine whether words or phrases can be detected, and such words or phrases are converted into VTT data.

[0083] At 810, one or more processors determine whether the voice command results in an action of the program. In one example, the VTT data is compared with a list of program control points. The program control points can be determined based on historical data and information including information generated by machine learning. In another example, the application control points compared with the VTT include each application control point associated with the screen. Alternatively, in another example, only a subset of the control points based on the gaze window is utilized. In particular, although additional control points are available, only the control points within the gaze window are compared with the VTT data. If no match or partial match is performed in the comparison, no additional action occurs.

[0084] If at 810, it is determined by one or more processors that a match or partial match is made, then at 812, the control point is executed. For example, if a user of the document provides a command "check spelling and grammar" that is placed into the VTT data and the list includes "spelling and grammar", then the control point for spelling and grammar checking begins operation. In one example, a driver running on a device such as a user input device (e.g., a mouse) can scroll to the corresponding coordinates of the spelling and grammar checking position on the display and generate a selection event (e.g., clicking on the spelling and grammar checking function) to the operating system in real time. In this way, the document is automatically spelled and grammar checked in real time without having to go through a menu.

[0085] Fig. 9 A block diagram of a method 900 for controlling an application during a conference call is shown. The method 900 may be implemented using Figures 1 to 7 In particular, the method can utilize context data and context applications to determine whether a user of a teleconferencing application attempts to communicate with others on a call when a mute function is provided.

[0086] At 902, one or more processors of the primary electronic device obtain context data related to the user. One or more processors may include a context application that obtains context data from a first sensor, obtains context data from a secondary electronic device, and the like. Context data may include data obtained using methods such as hearing, vision, touch, infrared, temperature, etc. In one example, a user may be determined as an individual closest to a sensor of a camera device that is a primary electronic device. In another example, facial recognition software may be used to identify the user of the primary electronic device. In one example, the context data includes eye gaze data and head position data obtained from a CV method as described herein. In other examples, sensors may detect movement of a user's mouth, whether the user is viewing a workspace, or other actions that may be used to determine an expected voice command.

[0087] At 904, it is determined whether the user is muted during the conference call. If the conference call application is not muted, no further action is taken. Alternatively, if the user is muted, then at 906, one or more processors analyze contextual data to identify the user's intent to speak. In particular, input from one or more sensors such as a camera, microphone, infrared, ultrasound, etc. can be used to detect when the user appears in front of the camera and when the user intends to speak to the primary electronic device.

[0088] Lookup tables, decision trees, mathematical models, computer-generated models, artificial intelligence algorithms, and the like may be used to determine the user's intention to speak based on contextual data. For example, the software and hardware may operate on a primary electronic device that uses any method including contextual data from a camera, ultrasound, infrared, CV, and the like to identify whether a user is present. In addition, sound detection hardware and software including a microphone may be used to determine whether sound is received from a user. Additional filtering may also be applied to detect the distance of the sound from the source to the microphone, whether the sound is human speech, whether the sound is made by the user of the primary electronic device, and the like. For example, when someone appears in the field of view of a camera that is speaking, human voice confirmation may be provided. In particular, it may be determined, for example, by CV that the user is moving their mouth.

[0089] Once a confirmation that the user is speaking to the primary electronic device is provided, additional contextual data can be used to determine the user's intention. For example, CV can be used to detect whether the user is viewing the display in the workspace of the primary electronic device or the user with the teleconference application. In addition, the window on one or more displays that the user is watching can be analyzed to obtain information related to the user's intention. In another example, the analyzed contextual data can include whether the user is located in their workspace, whether other people are in the workspace, etc. For this reason, during a teleconference, other people, animals, etc. that are not users can be filtered out from the picture and sent to other people on the teleconference. In another example, CV can determine the user's posture and gaze position, and to determine the focus of the user on the display. Nevertheless, regardless of the contextual data and analysis processing, one or more processors use the obtained contextual data to determine the user's intention and whether the user expects to speak to others during the call.

[0090] If at 906, the user is not determined to have an intent to speak, then no further action occurs. For example, if the user looks away from the display and is talking to another person in the environment about something unrelated to the meeting, no action occurs. If the sound is made by a barking dog in the environment, no action is taken. If the user is looking down at another electronic device to watch a sporting event and make comments when the camera is off, no action is taken. Thus, when the user has no intent to speak, the microphone remains off during the conference call, thus enabling worry-free meetings during online meetings.

[0091] Alternatively, if it is determined at 906 that the user has an intent to speak, then the one or more processors automatically turn on or activate the microphone of the teleconferencing application in real time at 908. Such automatic activation eliminates the need for the user to have to press a key, scroll a pointer to a button with a mouse, have to remember that they are currently muted, etc. Thus, meetings are more efficient and less frustrating for the user.

[0092] As will be appreciated, various aspects may be implemented as systems, methods, or computer (device) program products. Thus, various aspects may take the form of a complete hardware implementation or an implementation including hardware and software, both of which may be generally referred to herein as "circuits," "modules," or "systems." Additionally, various aspects may take the form of a computer (device) program product implemented in one or more computer (device) readable data storage devices having computer (device) readable program code implemented thereon.

[0093] Any combination of one or more non-signal computer (device) readable media may be utilized. The non-signal medium may be a data storage device. The data storage device may be, for example, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing. More specific examples of data storage devices may include a portable computer floppy disk, a hard disk, a random access memory (RAM), a dynamic random access memory (DRAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0094] The program code for performing the operation can be written in any combination of one or more programming languages. The program code can be executed completely on a single device, partially on a single device, executed as an independent software package, partially on a single device and partially on another device, or executed completely on another device. In some cases, the device can be connected by any type of network--including a local area network (LAN) or a wide area network (WAN), or can be connected by other devices (for example, by using the Internet of an Internet service provider) or by a hard-wired connection such as by a USB connection. For example, a server with a first processor, a network interface and a storage device for storing code can store a program code for performing the operation, and the code is provided to a second device with a second processor for executing code on a second device by a network interface via a network.

[0095] Various aspects are described herein with reference to the accompanying drawings, which illustrate example methods, devices, and program products according to various example implementations. These program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device or information processing device to produce a machine, so that the instructions executed by the processor of the device implement the specified function / action. The program instructions may also be stored in a device-readable medium, which may instruct the device to function in a particular manner, so that the instructions stored in the device-readable medium produce an article including instructions for implementing the specified function / action. The instructions may also be loaded onto the device so that a series of operating steps are performed on the device to produce a process implemented by the device, so that the instructions executed on the device provide a process for implementing the specified function / action.

[0096] The unit / module / application herein may include any processor-based or microprocessor-based system, including systems using microcontrollers, reduced instruction set computers (RISC), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), logic circuits, and any other circuits or processors capable of performing the functions described herein. Additionally or alternatively, the module / controller herein may represent a circuit module, which may be implemented as hardware having associated instructions (e.g., software stored on tangible and non-transient computer-readable data storage devices such as computer hard drives, ROMs, RAMs, etc.) for performing the operations described herein. The above examples are exemplary only, and are therefore not intended to limit the definition and / or meaning of the term "controller" in any way. The unit / module / application herein may execute a set of instructions stored in one or more storage elements in order to process data. The storage element may also store data or other information as desired or required. The storage element may be in the form of an information source or physical memory element within the module / controller herein. The instruction set may include various commands that instruct the module / application herein to perform specific operations of methods and processing of various embodiments of the subject matter described herein. The instruction set may be in the form of a software program. The software may be in various forms such as system software or application software. Furthermore, the software may be in the form of a collection of separate programs or modules, a program module within a larger program, or a portion of a program module. The software may also include modular programming in the form of object-oriented programming. The processing of input data by a processing machine may be in response to user commands, or in response to the results of previous processing, or in response to a request made by another processing machine.

[0097] It should be understood that the subject matter described herein is not limited in its application to the details of the construction and arrangement of the components set forth in the description herein or shown in the drawings herein. The subject matter described herein can have other embodiments and can be practiced or executed in various ways. In addition, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of "including", "comprising" or "having" and variations thereof herein is intended to cover the items listed thereafter and their equivalents and additional items.

[0098] It should be understood that the above description is intended to be illustrative, not restrictive. For example, the above embodiments (and / or aspects thereof) can be used in combination with each other. In addition, without departing from its scope, many modifications can be made to adapt specific situations or materials to the teachings herein. Although the types of sizes, coatings and materials described herein are intended to define various parameters, they are by no means restrictive and are illustrative in nature. After reviewing the above description, many other embodiments will be obvious to those skilled in the art. Therefore, the scope of the embodiments should be determined with reference to the full scope of the equivalents given by the attached claims and these claims. In the attached claims, the terms "including" and "in which" are used as the plain English equivalents of the corresponding terms "comprising" and "wherein". In addition, in the attached claims, the terms "first", "second" and "third" etc. are used only as labels and are not intended to impose numerical requirements on their objects or the execution order of their actions.

Claims

1. A system for operating a program on a primary electronic device, comprising: A main electronic device having a memory for storing executable instructions and one or more processors, wherein the one or more processors perform the following operations when implementing the executable instructions: obtaining contextual data associated with a user of the primary electronic device; determining a first location on a display of the primary electronic device based on the contextual data; determining a voice command associated with the first location on the display; as well as A control point at the first location is actuated based on the voice command.

2. The system according to claim 1, wherein: To determine the voice command, the one or more processors are further configured to: detecting sounds in the user's environment; determining that the user created the sound; recognizing at least one word from the sound; converting the at least one word into speech-to-text data; as well as The at least one word is compared to a list of words associated with the first location to determine the voice command.

3. The system according to claim 1, wherein: The one or more processors are further configured to determine a second location on the display in response to actuating the control point at the first location.

4. The system according to claim 1, wherein: The one or more processors are further configured to: determining whether data defining the control point is stored in the memory; and The control points in the memory are dynamically adjusted in response to determining that the control points are in the memory.

5. The system according to claim 1, wherein: The primary electronic device includes at least one sensor in communication with the one or more processors, and the one or more processors obtain the contextual data from the at least one sensor.

6. The system according to claim 1, wherein: The one or more processors utilize computer vision to determine the first location on the display.

7. The system according to claim 1, wherein: The one or more processors are also configured to obtain contextual data from communications from a secondary electronic device.

8. The system of claim 1, the one or more processors further configured to determine that the control point is within the first position prior to actuating the control point.

9. The system according to claim 1, wherein: Determining the voice command associated with the first location includes analyzing the contextual data using an artificial intelligence application.

10. A method comprising: under the control of one or more processors configured with executable instructions; obtaining contextual data associated with a user of a primary electronic device; determining a first location on a display of the primary electronic device based on the contextual data; determining a voice command associated with the first location on the display; as well as A control point at the first location is actuated based on the voice command.

11. The method according to claim 10, wherein: Determining the voice command includes: detecting sounds in the user's environment; determining that the user created the sound; recognizing at least one word from the sound; converting the at least one word into speech-to-text data; and The at least one word is compared to a list of words associated with the first location to determine the voice command.

12. The method of claim 10, further comprising determining a second position on the display in response to actuating the control point at the first position.

13. The method according to claim 10, further comprising: determining whether data defining the control point is stored in a memory; as well as The control points in the memory are dynamically adjusted in response to determining that the control points are in the memory. The method of claim 10 , further comprising obtaining context data from communications from a secondary electronic device.

15. The method of claim 10, further comprising determining that the control point is within the first position prior to actuating the control point.

16. The method according to claim 10, wherein: Determining the voice command associated with the first location includes analyzing the contextual data using an artificial intelligence application.

17. A computer program product comprising a non-transitory computer readable storage medium comprising computer executable code for: obtaining contextual data associated with a user of a primary electronic device; determining a first location on a display of the primary electronic device based on the contextual data; determining a voice command associated with the first location on the display; as well as A control point at the first location is actuated based on the voice command.

18. The computer program product of claim 17, wherein: The computer executable code is also used to: detecting sounds in the user's environment; determining that the user created the sound; recognizing at least one word from the sound; converting the at least one word into speech-to-text data; as well as The at least one word is compared to a list of words associated with the first location to determine the voice command.

19. The computer program product of claim 17, wherein: The computer executable code is also for determining a second location on the display in response to actuating the control point at the first location.

20. The computer program product of claim 17, wherein: The computer executable code is also used to: determining whether data in the memory defines the control point; and The control points in the memory are dynamically adjusted in response to determining that the control points are in the memory.

21. A system for operating a program on a primary electronic device, comprising: A main electronic device having a memory for storing executable instructions and one or more processors, wherein the one or more processors perform the following operations when implementing the executable instructions: obtaining contextual data associated with a user of the primary electronic device; determining, based on the contextual data, whether the program is muting the user; In response to determining that the program is muting the user, determining based on the contextual data whether the user intends to communicate using voice via the program; as well as When it is determined that the user intends to communicate using the voice, the program is automatically actuated to unmute to allow the user's voice to be transmitted to the program.

22. The system of claim 21, wherein: The program in question is a teleconferencing application.

23. The system of claim 21, wherein: To determine that the user intends to communicate using voice, the one or more processors are configured to: analyzing the contextual data to determine that the user is in front of the primary electronic device; as well as Computer vision CV is used to determine that the user's gaze is viewing a workspace.

24. The system of claim 21, further comprising: A sensor is coupled to the one or more processors and configured to obtain the context data.

25. The system of claim 24, wherein: The sensor is at least one of a camera, a microphone, an infrared sensor or a temperature sensor.

26. The system of claim 21, wherein: The one or more processors are further configured to: determining a workspace of the user; and A determination is made as to whether the user is within the workspace in front of the primary electronic device.

27. The system of claim 26, wherein: The one or more processors are further configured to: identifying another individual or animal within the workspace; and The other individual or the animal is filtered from the image data obtained by the one or more processors.

28. The system of claim 21, wherein: The one or more processors are further configured to: determining that the sound of the user is a voice command; and Implementing the voice command.

Images