Method and device for controlling graphic interfaces via voices

A graphical interface and voice control technology, applied in voice analysis, voice recognition, voice input/output, etc., can solve problems such as difficult to cover program execution path, poor support, large amount of calculation, etc., to improve usability and user experience effects

Active Publication Date: 2018-01-19
SAMSUNG ELECTRONICS CHINA R&D CENT +1
View PDF3 Cites 24 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] At present, most voice assistant software cannot support a large number of third-party applications well, and can only do some simple opening operations, and cannot operate every graphical interface in the third-party program
[0004] Some of the existing solutions need to manually pre-define the mapping relationship library of voice and operation, which is very costly and difficult to cover all the execution paths of the program; some require real-time image analysis of the entire screenshot to find Manipulating the area and recognizing the text in it requires a lot of calculations, making it difficult to provide a smooth user experience

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for controlling graphic interfaces via voices
  • Method and device for controlling graphic interfaces via voices
  • Method and device for controlling graphic interfaces via voices

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0045] see figure 1 , figure 1 It is a schematic flow chart of the voice control graphical interface in Embodiment 1 of the present application. The specific steps are:

[0046] Step 101, the terminal receives a voice command input by a user, and recognizes the voice command as a voice text.

[0047] Step 102, the terminal obtains the thumbnail of the screenshot of the current interface, and sends the thumbnail and the identifier of the currently used APP to the page layout server, so that the page layout server matches and corresponds to the information of the thumbnail and the identifier of the APP page layout information.

[0048] In actual implementation, the two operations of recognizing the voice command as voice text and obtaining the thumbnail of the screenshot and sending it to the page layout server can be performed at the same time, regardless of the sequence.

[0049] Step 103: When receiving the page layout information responded by the page layout server, the ...

Embodiment 2

[0052] see figure 2 , figure 2 It is a schematic flow chart of the voice control graphical interface in Embodiment 2 of the present application. The specific steps are:

[0053] Step 201, the terminal receives a voice command input by a user, and recognizes the voice command as a voice text.

[0054] Step 202, the terminal processes the voice text to obtain positioning description information, operation type and operation content of the target operation area.

[0055] In this step, it is necessary to perform natural language processing on the speech text, and then obtain the positioning description information, operation type and operation content of the target operation area.

[0056]The location description information of the target operation area is used to match the target operation area in the page layout information. If item B is selected, the information of the location description is option B, and the determined target operation area is the area where option B is ...

Embodiment 3

[0065] see image 3 , image 3 It is a schematic flow chart of the voice control graphical interface in Embodiment 3 of the present application. The specific steps are:

[0066] Step 301, the terminal receives a voice command input by a user, and recognizes the voice command as a voice text.

[0067] In step 302, the terminal obtains a thumbnail of a screenshot of the current interface, and determines whether the corresponding page layout information is locally stored according to the information of the thumbnail and the identifier of the currently used APP, and if so, executes step 303; otherwise, executes step 304.

[0068] Step 303, the terminal acquires locally stored matching page layout information. Execute step 307.

[0069] Step 304, the terminal sends the thumbnail image and the currently used APP identifier to the page layout server, so that the page layout server matches the corresponding page layout information according to the information of the thumbnail imag...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a method and device for controlling graphic interfaces via voices. The method comprises the following steps of: receiving a voice instruction input by a user and recognizing thevoice instruction as a voice text; obtaining a thumbnail of a screenshot of the current interface, and sending the thumbnail and an identifier of the currently used APP to a page layout server to ensure that the page layout server matches corresponding page layout information according to the thumbnail and the identifier of the APP; when the page layout information responded by the page layout server is received, using the voice text to match a target operation area corresponding to the voice instruction in the page layout information; and executing an operation appointed by the voice text inthe target operation area. The method and device are capable of improving the APP usability and user experience.

Description

technical field [0001] The invention relates to the technical field of speech recognition, in particular to a method and device for controlling a graphic interface by speech. Background technique [0002] With the development of speech recognition and natural language processing technology, voice assistant software allows us to control programs to complete operations through voice input. Generally, through additional development work, users can use complex voice commands to call the built-in APP or the open interface of commonly used APP, so that a series of operations can be performed at one time. [0003] At present, most voice assistant software cannot support a large number of third-party applications very well, and can only do some simple opening operations, and cannot operate every graphical interface in the third-party program. [0004] Some of the existing solutions need to manually pre-define the mapping relationship library of voice and operation, which is very co...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F3/16G10L15/22
Inventor魏涛武永超方华
OwnerSAMSUNG ELECTRONICS CHINA R&D CENT