Voice control method and electronic equipment

Abstract

The invention provides a voice control method and electronic equipment, relates to the technical field of electronics, and is used for enabling the electronic equipment to respond to a voice instruction of a user in a single-hand mode and execute simulated click operation at an accurate position. The method is applied to the electronic equipment with a voice control function and a single-hand operation function, and comprises the following steps: responding to a single-hand mode trigger event, and switching a full-screen display picture to a small-screen display picture in a single-hand mode display state; after the voice control function is started, performing voice recognition on the acquired audio data based on the voice control function; and if the voice recognition result indicates that the operation corresponding to the audio data belongs to the simulated click operation, determining a simulated click position of the simulated click operation in the small-screen display picture. If the simulation click operation is the element click operation, executing a task triggered by an interface element corresponding to the element click operation; and if the simulation click operation is the sliding operation, sliding the display picture from the simulation click starting position to the simulation click ending position.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of electronic technology, and in particular to a voice control method and electronic equipment. Background Art

[0002] In many scenarios, users may want to use one hand to operate electronic devices, such as using a mobile phone. However, some mobile phones are large in size, so users cannot click all areas on the screen with one hand. In this case, some mobile phones provide a one-handed mode, which reduces the display on the screen so that users can more conveniently use one hand to click any area of ​​the display.

[0003] In addition, many mobile phones can also enable users to control electronic devices through voice. In some scenarios, the user's voice command content is to control the mobile phone to perform a simulated click operation. In order for the mobile phone to perform a simulated click operation in response to the voice command, the position to be clicked is usually determined first, and then the simulated click is performed at the position.

[0004] Since the position of the displayed content on the screen of the mobile phone changes in the one-handed mode, when the user performs a simulated click operation by voice control of the mobile phone, the simulated click position determined by the mobile phone may be inaccurate. Thus, when the mobile phone performs a simulated click operation in response to a voice command, the click position error may occur. Summary of the invention

[0005] The embodiments of the present application provide a voice control method and an electronic device, which are used in a one-handed mode so that the electronic device can respond to a user's voice command and perform a simulated click operation at an accurate position.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, a voice control method is provided, the method using an electronic device having a voice control function and a one-hand operation function, the method comprising:

[0008] In response to a one-handed mode triggering event, the electronic device switches from a full-screen display to a small-screen display in a one-handed mode display state. After the voice control function is started, audio data can be collected based on the voice control function, and voice recognition is performed on the audio data collected by the electronic device to obtain a voice recognition result. When the voice recognition result indicates that the operation corresponding to the audio data belongs to a simulated click operation, the electronic device can determine the simulated click position of the simulated click operation in the currently displayed small-screen display. Among them, the simulated click operation can be an element click operation or a sliding operation. Specifically, when the simulated click operation is a sliding operation, the simulated click position can be divided into a simulated click starting position and a simulated click ending position. Finally, according to the simulated click position, the simulated click operation is performed.

[0009] In this solution, if the simulated click operation is an element click operation, the electronic device performs the task triggered by the interface element corresponding to the element click operation. If the simulated click operation is a sliding operation, the electronic device slides the display screen from the simulated click start position to the simulated click end position. In this way, it can be ensured that in the one-handed mode, in the scenario where the user uses voice to control the electronic device to perform a simulated click operation, the electronic device responds to the voice instruction of the simulated click, can perform the simulated click operation at the correct position, trigger the corresponding response operation, and avoid the problem of simulated click errors. On the premise that the electronic device accurately recognizes the user's voice instruction, the electronic device can respond to the voice instruction of the simulated click and accurately perform the simulated click operation.

[0010] In a possible implementation of the first aspect, the above-mentioned determination of the simulated click position corresponding to the simulated click operation in the small-screen display picture may specifically include: determining a first coordinate of the simulated click position in the screen coordinate system. In this implementation, before performing the simulated click operation according to the simulated click position, the above-mentioned method further includes: performing a first mapping process on the first coordinate to obtain a second coordinate in the screen coordinate system; wherein the first mapping process is used to map the coordinates of the point from the full-screen display picture to the small-screen display picture.

[0011] Further, in this embodiment, the electronic device performs a simulated click operation according to the simulated click position, which may specifically include: generating a simulated click event based on the second coordinate. Sending a simulated click event to the multi-mode input framework of the electronic device. In response to the simulated click event, the multi-mode input framework performs a second mapping process on the second coordinate. Among them, the second mapping process is used to map the coordinates of the point from the small-screen display to the full-screen display; it can be seen that the second mapping process and the first mapping process are two opposite mapping processes. Therefore, the multi-mode input framework performs a second mapping process on the second coordinate, and the first coordinate can still be obtained. Finally, the electronic device can perform a simulated click operation on the position of the first coordinate based on the multi-mode input framework. In this way, it is ensured that the electronic device finally performs the simulated click operation at the position of the simulated click position in the small-screen display, avoiding the problem of the position error of the simulated click operation.

[0012] In a possible implementation of the first aspect, the first mapping process is performed on the first coordinate to obtain the second coordinate in the screen coordinate system, which may specifically include: obtaining a full-screen display size of the electronic device in a full-screen display picture, a small-screen display size in a small-screen display picture, and a first mapping relationship from the full-screen display size to the small-screen display size. Afterwards, the coordinates of the point may be mapped from the full-screen display picture to the small-screen display picture according to the first mapping relationship.

[0013] Before the multi-mode input framework of the electronic device performs the simulated click operation in the one-handed mode, the coordinates of the point will automatically default to the coordinates of the small screen display screen, and map it to obtain the coordinates of the full screen display screen, and finally perform the simulated click operation for the coordinates of the mapping process. Therefore, in this embodiment, in order to make the position where the electronic device finally performs the simulated click operation still the coordinates in the small screen display screen, the first coordinates of the simulated click position of the acquired small screen display screen are pre-mapped to obtain the second coordinates. After that, the multi-mode input framework of the electronic device obtains the second coordinates, uses the second coordinates as the coordinates in the small screen display screen, and performs the second mapping process on them, and performs the simulated click operation on the coordinates obtained by the second mapping process. Since the second coordinates are obtained by the first mapping process of the first coordinates, and the second mapping process is the opposite mapping process to the first mapping process, therefore, after the multi-mode input framework of the electronic device performs the second mapping process on the second coordinates, the obtained coordinates are still the first coordinates, that is, the position of the simulated click position in the small screen display screen. In this way, it is ensured that the electronic device can perform the simulated click operation at the correct position.

[0014] In a possible implementation of the first aspect, the above-mentioned acquisition of the full-screen display size of the electronic device in the full-screen display picture, the small-screen display size in the small-screen display picture, and the first mapping relationship of transforming the full-screen display size to the small-screen display size may specifically include: acquiring the first width and first height of the full-screen display size, and the second width and second height of the small-screen display size. Based on the first height and the second height, determining the height reduction ratio of transforming the full-screen display size to the small-screen display size. Based on the first width and the second width, determining the width reduction ratio of transforming the full-screen display size to the small-screen display size. Finally, determine the first mapping relationship based on the height reduction ratio and the width reduction ratio.

[0015] In a possible implementation of the first aspect, performing the second mapping process on the second coordinate to obtain the first coordinate may specifically include: obtaining a second mapping relationship for transforming the electronic device from a small-screen display size to a full-screen display size. Based on the second mapping relationship, performing the second mapping process on the second coordinate to obtain the first coordinate.

[0016] In a possible implementation of the first aspect, when the simulated click operation is an element click operation, the above-mentioned determination of the simulated click position corresponding to the simulated click operation in the small-screen display may specifically include: analyzing the preset element corresponding to the element click operation. Then, searching the position corresponding to the preset element in the small-screen display as the simulated click position. In this solution, after determining the element click operation of the preset element corresponding to the simulated click operation, searching the position of the preset element in the small-screen display as the simulated click position can ensure that the simulated click position is a position in the small-screen display.

[0017] In a possible implementation of the first aspect, when the simulated click operation is an element click operation, determining the simulated click position corresponding to the simulated click operation in the small-screen display may specifically include: obtaining all elements contained in the small-screen display, and the position information corresponding to each element. Analyze the preset elements corresponding to the element click operation, and find the position corresponding to the preset elements from the position information. In this solution, all elements in the small-screen display and their corresponding positions may be obtained first. After determining that the operation corresponding to the audio data is an element click operation, find the position of the preset element from the position information of all elements to ensure that the simulated click position is in the small-screen display.

[0018] In a possible implementation of the first aspect, the method further includes: while executing the simulated click operation, displaying a simulated click track at a position corresponding to the simulated click operation, so that the user can more intuitively see the position where the simulated click operation is executed.

[0019] In a possible implementation manner of the first aspect, the aspect ratio of the full-screen display picture is the same as the aspect ratio of the small-screen display picture.

[0020] In a second aspect, an electronic device is provided, comprising: a processor, a memory, a display screen, and a recording device; the memory, the display screen, and the recording device are respectively coupled to the processor. The display screen is used to display the interface of the electronic device, and the recording device is used to collect audio data; the memory is used to store computer-executable instructions, and when the electronic device is running, the processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the voice control method as described in any one of the above-mentioned first aspects.

[0021] In a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer can execute the voice control method of any one of the above-mentioned first aspects.

[0022] In a fourth aspect, a computer program product comprising instructions is provided, which, when executed on an electronic device, enables the electronic device to execute any one of the voice control methods of the first aspect.

[0023] In a fifth aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting an electronic device to implement the functions involved in the first aspect above. In one possible design, the device also includes a memory for storing program instructions and data necessary for the electronic device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.

[0024] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a scenario in which a user uses voice to control an electronic device provided in an embodiment of the present application;

[0026] Figure 2A A schematic diagram of a mobile phone setting interface provided in an embodiment of the present application;

[0027] Figure 2B A schematic diagram of an interface for controlling a mobile phone to perform operations by voice provided in an embodiment of the present application;

[0028] Figure 2C A schematic diagram of an interface for controlling a mobile phone to perform operations by voice provided in an embodiment of the present application;

[0029] Figure 3AA schematic diagram of a mobile phone setting interface provided in an embodiment of the present application;

[0030] Figure 3B A schematic diagram of a process of triggering a one-handed mode provided in an embodiment of the present application;

[0031] Figure 3C A schematic diagram of an interface for voice-controlled mobile phone operations in one-handed mode provided in an embodiment of the present application;

[0032] Figure 3D A schematic diagram of an interface for voice-controlled mobile phone operations in one-handed mode provided in an embodiment of the present application;

[0033] Figure 4A A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0034] Figure 4B A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0035] Figure 5A A flow chart of a voice control method provided in an embodiment of the present application;

[0036] Figure 5B A schematic diagram of a mobile phone interface in one-handed mode provided in an embodiment of the present application;

[0037] Figure 5C A schematic diagram of a coordinate mapping provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of a coordinate mapping provided in an embodiment of the present application;

[0039] Figure 7 A flow chart of a voice control method provided in an embodiment of the present application;

[0040] Figure 8 A structural block diagram of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] With the development of technology, it is now possible for users to control electronic devices through voice.

[0042] For example, when a user is cooking in the kitchen, watching a video, or eating or watching a TV show, both hands of the user may be occupied. In this case, the user can usually control the electronic device to perform corresponding operations by speaking voice commands. The electronic device collects the user's voice through a microphone, analyzes and recognizes the user's voice, and executes the command corresponding to the user's voice, so that the user can control the electronic device through voice. Figure 1As shown, the user 10 can control the mobile phone 1 to perform corresponding operations by speaking voice commands within a certain range close to the mobile phone 1.

[0043] Generally speaking, in order to save power consumption of electronic devices and avoid false triggering, the voice control function needs to be turned on before it can be used. For example, the user inputs a preset word (called a wake-up word) to the electronic device through voice to wake up the electronic device. After the electronic device is awakened, it can execute the instructions corresponding to the user's voice, that is, the voice control function is turned on. Alternatively, the user can turn on or off the voice control function by turning on or off the preset switch in the human-computer interaction interface of the electronic device.

[0044] In different electronic devices, the voice control function may have different names, such as "voice control", "intelligent voice", "voice assistant", "see and speak", "voice command", "free command", "intelligent AI", etc. The specific implementation of voice control functions with different names may also be different.

[0045] The following is a brief description of the relevant information that can be seen and said:

[0046] It can be seen that it is implemented locally by electronic devices and does not require a network connection.

[0047] Visible and Talkable is controlled by a preset switch. Users can turn on Visible and Talkable by turning on the preset switch, and can exit Visible and Talkable by turning off the preset switch.

[0048] Taking the electronic device as a mobile phone as an example, for example, Figure 2A As shown, the user can open the settings function of the mobile phone 1; for example, the user clicks on the application icon of the "Settings" application on the desktop. In response to the user clicking on the application icon of the "Settings" application, the mobile phone 1 displays the "Settings" interface 101. The "Settings" interface 101 includes a "Smart Voice" option 102, and the "Smart Voice" option 102 is used to set the smart voice function. Exemplarily, in response to the user clicking on the "Smart Voice" option 102, the mobile phone 1 displays the "Smart Voice" interface 103, and the "Smart Voice" interface 103 includes a "Visible and Speakable" option 104. The user can click on the "Visible and Speakable" option 104 to set options related to the Visible and Speakable function. Exemplarily, refer to Figure 2AIn response to the user's click operation on the "See and Say" option 104, the mobile phone 1 displays the "See and Say" interface 105. Optionally, the "See and Say" interface 105 includes a prompt message 106 for prompting the user how to use the "See and Say" function. The "See and Say" interface 105 also includes a "See and Say" switch 107. The user can click the "See and Say" switch 107 to turn on or off the "See and Say" switch. In one example, in response to receiving the user's click operation on the "See and Say" switch 107, the "See and Say" switch of the mobile phone 1 is turned on to turn on the "See and Say" function. Optionally, the "See and Say" interface 105 displays a prompt message 108 for prompting the user that the "See and Say" function has been successfully turned on.

[0049] In one embodiment, after the visible-to-speak function is turned on, the mobile phone 1 displays a first prompt icon, which indicates that the visible-to-speak function has been turned on. Figure 2A As shown, after the “see and speak” switch 107 is turned on, the status bar of the display interface of the mobile phone 1 displays a prompt icon 10a, indicating that the see and speak function has been turned on.

[0050] In one scenario, the preset switch corresponding to Visible and Talking on the electronic device is not turned on, and the microphone of the electronic device is not turned on. When the preset switch corresponding to Visible and Talking is turned on, the electronic device starts the microphone and starts the corresponding recording channel of Visible and Talking in the system and driver. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talking application for processing through the recording channel corresponding to Visible and Talking, so that the user can control the electronic device through voice.

[0051] In another scenario, the preset switch corresponding to the visible and speaking function of the electronic device is not turned on, and the microphone of the electronic device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. When the preset switch corresponding to the visible and speaking function is turned on, the corresponding recording channel of the visible and speaking function is started in the system and driver of the electronic device. In this way, the voice (audio stream) collected by the microphone can be sent to the visible and speaking application for processing through the recording channel corresponding to the visible and speaking function, so that the user can control the electronic device through voice.

[0052] After Visible is Talking is turned on, the corresponding recording channel of Visible is Talking is started in the system and driver of the electronic device, and the electronic device enters a "long reception" state to continuously collect surrounding sounds. Users can give commands to the electronic device at any time through voice without having to enter a wake-up word to wake up the electronic device.

[0053] In one implementation, when any function on the electronic device turns on the voice recording function of the electronic device (turns on the microphone and establishes a recording channel), the electronic device will send a prompt message to the user to remind the user that the electronic device is in a voice collection state. In this way, the user's privacy can be prevented from being leaked. For example, after the visible and audible function is turned on, the electronic device enters a state of continuous voice collection, and a second prompt icon is displayed on the display interface of the electronic device. The second prompt icon indicates that the recording channel is turned on, which is used to remind the user that the microphone is collecting voice. For example, Figure 2A As shown, the status bar of the display interface of the mobile phone 1 displays a prompt icon 10b, indicating that the recording channel is turned on, which is used to prompt the user that the microphone is collecting voice.

[0054] After the See and Speak function is turned on, the electronic device establishes a recording channel and continuously collects the user's voice through the microphone. The user can input voice into the electronic device at any time. The electronic device analyzes and recognizes the voice input by the user and executes the instructions corresponding to the user's voice.

[0055] In some embodiments, the voice commands supported by Visible and Speakable can be divided into system commands and application commands. Among them, system commands can be used in any interface, and application commands support use on the application interface displayed after some applications are started. Exemplarily, system commands may include: return to desktop, return, open application, slide up, slide down, slide left, slide right, brightness adjustment and volume adjustment, etc. Application commands may include: play, pause, full screen, like, favorite, next, previous, fast forward and rewind, etc. It should be noted that the above commands are some examples of commands supported by the Visible and Speakable voice control function, that is, when the Visible and Speakable voice control function is started, the user can control the mobile phone 1 to perform the operations corresponding to the above voice commands by speaking the corresponding voice commands.

[0056] For the same purpose of controlling mobile phone 1, the voice commands spoken by different users may not be exactly the same. Therefore, each of the above voice commands can correspond to multiple different expressions. For example, for the system voice command of returning to the desktop, the user can say any of the voice expressions such as "desktop", "return to desktop", "return to desktop", "return to home screen", etc. to control mobile phone 1 to return to the desktop. Accordingly, when mobile phone 1 detects the above voice expression, it can recognize it as a voice command to return the current page to the desktop.

[0057] In one example, if Figure 2B As shown, the mobile phone 1 displays a home screen interface 201. The user inputs a voice message "Open calendar" to the mobile phone 1. In response to receiving the voice message "Open calendar" input by the user, the mobile phone 1 starts the calendar application and displays a user interface 202 of the calendar application.

[0058] In another example, Figure 2C As shown, mobile phone 1 displays an interface 203 of a short video application. The user inputs a voice message "pause" to mobile phone 1. In response to receiving the voice message "pause" input by the user, mobile phone 1 executes the command corresponding to the voice message "pause" and displays interface 204. A "play" button 205 is displayed on the interface 204 to prompt the user that the short video has been paused. The user can click the "play" button 205 to continue playing the short video. Alternatively, the user can also input a voice message "play" to mobile phone 1 to continue playing the short video.

[0059] Furthermore, on the above-mentioned display interface 203, the user can also control operations such as liking and collecting the currently playing short video through the visible and spoken voice control function.

[0060] In some embodiments, the specific implementation process of the mobile phone 1 performing the corresponding operation in response to the voice command can refer to the introduction in the relevant technology, which will not be repeated in the embodiments of the present application. Exemplarily, after determining that the audio data includes the voice command supported by the visible and audible voice control function, the specific operation corresponding to the voice command can be performed through the multi-modal input framework of the mobile phone 1.

[0061] In some embodiments, after the above-mentioned visible-can-speak voice control function is activated, the mobile phone 1 will obtain the text data displayed on the current page in real time, analyze and obtain the words contained therein, and the corresponding positions of each word and its related content (such as the application icon of the corresponding application) in the current page. Figure 2B In the main screen interface 201 shown, the mobile phone 1 can obtain the text data displayed on the current page and analyze the words contained therein, such as: "clock", "calendar", "gallery", "memo", "file management", "email" and "music" and so on. It should be noted that the specific implementation process of the mobile phone 1 obtaining the text data displayed on the current page and analyzing the words contained therein can refer to the description in the relevant technology and will not be repeated in the embodiments of the present application. Among them, the position corresponding to the word "calendar" on the current page may be the position where the word "calendar" itself is located; or, the position corresponding to the word "calendar" on the current page may also be the position where the display content related to "calendar" is located. Figure 2B In the main screen interface 201 shown, the display content related to "calendar" can be the location where the application icon of the "calendar" application is located; Figure 2B The area enclosed by a dotted line in the main screen interface 201 is shown.

[0062] It should be noted that the above-mentioned voice control function is only an example of a voice control function. In other embodiments, the voice control function may also be a voice control function. Exemplarily, the voice control function may also be: a voice assistant function that wakes up the voice control through a preset wake-up word, a free command function that wakes up the voice control through a preset scene, and so on.

[0063] In addition, in many scenarios, users may want to use a mobile phone with one hand. However, some mobile phones are large in size, which makes it impossible for users to click all areas on the screen with one hand. In this case, some mobile phones provide a one-handed operation function, which reduces the display screen on the screen so that users can more conveniently click any area on the display screen with one hand. It should be noted that the above-mentioned one-handed operation function can correspond to an application or a function in a certain application.

[0064] Please refer to Figure 3A , wherein the setting interface also includes a setting option 300 for the auxiliary function. In response to the user's triggering operation on the setting option 300 for the auxiliary function, the user can enter Figure 3A The setting interface 301 for one-handed operation is shown. A switch 302 that allows one-handed operation to be triggered can be set in the setting interface 301. The switch 302 is used to turn on or off whether the user is allowed to trigger the one-handed mode through a gesture. Specifically, after the switch 302 is turned on, the user can trigger the one-handed mode of the mobile phone 1 and enter the small-screen display state by performing a preset gesture on the screen. Among them, the preset gesture can be set according to actual conditions and is not limited in the embodiments of the present application. It should be noted that the above-mentioned one-handed operation function can correspond to an application or a function in a certain application.

[0065] For example, taking the switch 302 in the open state as an example, please refer to Figure 3B , please show the main screen interface 303 of mobile phone 1. In the display interface of mobile phone 1 displaying the main screen interface 303, the user performs a preset gesture, which will trigger the one-handed mode of mobile phone 1. That is, mobile phone 1 will call up a small screen, such as Figure 3B The reduced home screen interface 304 is shown. Figure 3B In the area 305 outside the reduced main screen interface 304, the mobile phone displays a prompt message: "One-handed mode, click outside the small screen area to exit." The user can exit the one-handed mode by clicking outside the reduced main screen interface 304, and the display interface will be restored to the full screen display size.

[0066] It is understandable that the reduced home screen interface 304 is closer to the right side of the screen, which is more suitable for the user to operate with the right hand. In other embodiments, after the user performs a preset gesture on the home screen interface 303, the mobile phone 1 can also enter the following Figure 3B The reduced home screen interface 306 is more suitable for the user to operate the mobile phone with his left hand. It should be noted that in other embodiments, after the mobile phone 1 is triggered to enter the one-handed mode, the size and location of the small screen display area can be adjusted according to the user's wishes to display in other sizes or at other locations.

[0067] Further, in some embodiments, the switching from the main screen interface 303 to the reduced main screen interface 304, or the switching from the main screen interface 303 to the reduced main screen interface 306, in which the display screen changes from the full screen display size to the small screen display size, is reduced in proportion. That is, the aspect ratio of the display screen in the small screen display state is the same as the aspect ratio of the display screen in the full screen display state.

[0068] It should be noted that the above-mentioned reduced main screen interface 304 and reduced main screen interface 306 are two display examples of the one-handed mode display state. In other embodiments, in the one-handed mode display state, the mobile phone 1 can also display other display screens, such as a half-screen display screen near the bottom of the screen.

[0069] Furthermore, the following is an example in which the user speaks “Open [Calendar]” in the case of the above-mentioned reduced main screen interface 304. Mobile phone 1 analyzes this voice content and determines that it corresponds to the voice instruction “Open [application]” supported by the visible and audible voice control function, and that it contains the word [Calendar]. At the same time, by obtaining the words contained in the reduced main screen interface 304, the mobile phone can determine that the reduced main screen interface 304 contains the word [Calendar]. At this time, in response to the voice instruction “Open [Calendar]”, mobile phone 1 will perform a simulated click operation on the position corresponding to the word [Calendar] and its corresponding control in the current small-screen display.

[0070] Specifically, in some embodiments, in response to the voice command "open [calendar]", the mobile phone 1 injects a simulated click event into the system (such as a multi-modal input framework), and the simulated click event carries the position D1 of [calendar] in the current small-screen display image, such as Figure 3C An area 307 is shown enclosed by a solid circle.

[0071] However, in the one-handed mode display state, for the position D1, the system will automatically convert the position D1 to the position D2 where the [calendar] is located in the full screen display size of the mobile phone. For example, D2 can be as follows Figure 2B The dotted line area (simulated click position) shown in the main screen interface 201 is shown, or D2 can also be in Figure 3CThis position corresponds to the dotted circle area 308. Then, the system performs a simulated click operation on the position D2. Figure 3C As shown, the area 308 encircled by the dotted line does not correspond to the location of the calendar application in the one-handed mode display state. Therefore, if the mobile phone 1 performs a simulated click operation in the area 308 encircled by the dotted line, the calendar application cannot be accurately opened.

[0072] Moreover, in this example, the area 308 encircled by the dotted line is outside the reduced main screen interface 304. Therefore, in response to the voice command, the mobile phone 1 performs a simulated click operation at the position corresponding to the area 308 encircled by the dotted line, which may cause the mobile phone 1 to exit the one-handed mode and enter the Figure 3C The main screen interface 309 is shown. That is, for the user, after the voice operation of "open [calendar]" is spoken, what is seen is that the mobile phone 1 exits the one-handed mode, rather than the mobile phone 1 opening the calendar application.

[0073] Alternatively, in other embodiments, the area 308 encircled by the dotted line is located at another location of the reduced main screen interface 304. After the simulated click operation is performed in response to the voice command, the display interface of the mobile phone 1 may not change, or the location of the icon of another application may be clicked, causing the mobile phone 1 to open another application. For the user, after the voice operation of "open [calendar]" is spoken, no change is seen in the display interface of the mobile phone 1. Alternatively, after the user speaks the voice operation of "open [calendar]", the user sees that the mobile phone 1 opens another application.

[0074] In other embodiments, the user speaks a voice content of "slide down" in the case of the above-mentioned reduced main screen interface 304 as an example for explanation. According to the voice content, the mobile phone 1 can determine that the voice content corresponds to the voice instruction "slide down" supported by the visible voice control function, and the operation corresponding to the voice instruction is a simulated click operation. In the process of performing the simulated click operation in response to the voice instruction, the mobile phone 1 needs to determine the simulated click starting position and the simulated click ending position. Then, according to the simulated click starting position and the simulated click ending position, the simulated sliding operation is performed. It can be understood that the sliding operation can be understood as a continuous clicking operation. Specifically, the mobile phone 1 can first perform a simulated pressing operation at the simulated click starting position, and then perform a simulated sliding operation from the simulated click starting position to the simulated click ending position, and finally perform a simulated lifting operation at the simulated click ending position to complete the simulated sliding operation. It should be noted that the specific implementation process of the mobile phone 1 determining the simulated click starting position and the simulated click ending position can refer to the introduction in the relevant technology, and will not be repeated in the embodiments of the present application.

[0075] From the above description, it can be seen that in the small screen display state of the one-handed mode, the system of the mobile phone 1 will automatically convert the position where the simulated click operation is required to be performed to the position in the full screen display size before performing the simulated click operation. Similarly, in the process of the mobile phone 1 performing the simulated sliding operation, for the position D3 of the simulated click start position and the position D4 of the simulated click end position determined by the mobile phone 1, the system will also automatically convert them to the position in the full screen display size of the mobile phone 1. Figure 3D As shown, after receiving the voice command "slide down", the mobile phone 1 can determine the simulated click start position D3 and the simulated click end position D4. Before performing the simulated sliding operation, the system converts the positions D3 and D4 to Figure 3D Positions D5 and D6 are shown. Figure 3D As shown in FIG. 1 , the position of D5 is outside the area of ​​the reduced main screen interface. Then, when the mobile phone 1 performs a simulated sliding operation and performs a simulated pressing operation at the simulated click start position, the system may mistakenly recognize it as a click operation outside the small screen, causing the mobile phone 1 to exit the one-handed mode. Figure 3D As shown, after the user says the voice command "swipe down", the mobile phone 1 may change from the reduced main screen display interface 310 to the full-screen main screen display interface 311. However, the effect corresponding to this response operation of the mobile phone 1 does not correspond to the voice command "swipe down" said by the user.

[0076] The above-mentioned situations all show that in the one-handed mode, when the user controls the mobile phone 1 through voice to perform a simulated click operation, the problem of the click position error may occur. When the simulated click position is wrong, it is possible that the operation performed by the mobile phone cannot match the user's intention, thereby causing inconvenience to the user. It should be noted that the voice instructions of "open [calendar]" and "swipe down" in the above-mentioned embodiment are only two examples of the voice instructions corresponding to the simulated click operation. In other embodiments, the voice instructions corresponding to the simulated click operation may also include other voice instructions, such as swipe up, swipe left, swipe right, etc. Among them, the operation to be performed corresponding to the voice instruction of "open [calendar]" can be divided into an element click operation, and the operations to be performed corresponding to the voice instructions of swipe up, swipe down, swipe left, and swipe right can be divided into sliding operations.

[0077] Based on this, the present application proposes a voice control method, which is applied to the scenario where the electronic device is in one-handed mode and the user uses voice to control the electronic device to perform an operation. Exemplarily, the electronic device can be the above-mentioned mobile phone 1. In the technical solution provided in the embodiment of the present application, when the mobile phone 1 detects that the corresponding operation in the audio data belongs to a simulated click operation, if it is determined that the current mobile phone 1 is in a small-screen display screen in one-handed mode, the simulated click position of the simulated click operation in the small-screen display screen is obtained. Then, the mobile phone 1 performs the simulated click operation according to the simulated click position. Specifically, in the case where the simulated click operation is an element click operation (such as the above-mentioned "open [calendar]"), the task triggered by the interface element corresponding to the element click operation is executed (such as opening the [calendar] application and displaying the calendar interface). In the case where the simulated click operation is a sliding operation (such as the above-mentioned "slide down"), the simulated click start position and the simulated click end position are obtained, and then the display screen is slid from the simulated click start position to the simulated end position. In this way, it can be ensured that in the scenario where the user uses voice to control the electronic device to perform a simulated click operation in one-handed mode, the mobile phone 1 responds to the voice instruction of the simulated click, can perform the simulated click operation at the accurate position, trigger the corresponding response operation, and avoid the problem of simulated click errors. On the premise that the mobile phone 1 accurately recognizes the user's voice command, the mobile phone 1 can respond to the voice command of the simulated click and accurately perform the simulated click operation.

[0078] The voice control method provided in the embodiment of the present application can be applied to an electronic device 100 including a microphone. The electronic device 100 may include a mobile phone, a tablet computer, a laptop computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a smart home device (e.g., a smart TV, a smart screen, a large screen, a smart speaker, a smart air conditioner, etc.), a personal digital assistant (PDA), a wearable device (e.g., a smart watch, a smart bracelet, etc.), a vehicle-mounted device, a virtual reality device, etc., and the embodiment of the present application does not impose any restrictions on this.

[0079] like Figure 4AThe figure shows a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. Exemplarily, the electronic device 100 may be the mobile phone 1 mentioned above. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, etc.

[0080] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0081] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, the processor 110 is used to execute the voice control method in the embodiment of the present application.

[0082] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0083] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0084] The USB interface 130 is an interface that complies with USB standard specifications, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transmit data between the electronic device 100 and a peripheral device.

[0085] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0086] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and an application required for at least one function (such as a sound playback function, an image playback function, etc.).

[0087] In addition, the internal memory 121 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0088] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 .

[0089] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0090] In some other embodiments, the power management module 141 may also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 may also be disposed in the same device.

[0091] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0092] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0093] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0094] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0095] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0096] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170 and the application processor.

[0097] The audio module 170 is used to convert digital audio signals into analog audio signals for output, and is also used to convert analog audio inputs into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110. The audio module 170 can include a speaker 170A and a microphone 170B.

[0098] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or conduct a video conference through the speaker 170A.

[0099] Microphone 170B, also known as a "microphone", is used to convert sound signals into electrical signals. When making a video call, video conference, or using a voice assistant, the user can speak by putting their mouth close to the microphone 170B to input the sound signal into the microphone 170B. The electronic device 100 can be provided with at least one microphone 170B. In other embodiments, the electronic device 100 can be provided with two microphones 170B, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170B to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0100] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A.

[0101] The touch sensor 180B is also called a "touch panel". The touch sensor 180B can be arranged on the display screen 194, and the touch sensor 180B and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180B is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180B can also be arranged on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0102] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0103] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0104] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0105] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.

[0106] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0107] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting or removing the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0108] The voice control methods in the following embodiments can all be implemented in the electronic device 100 having the above hardware structure.

[0109] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0110] Figure 4B It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0111] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, application framework layer, Android runtime (ART) and native C / C++ library, hardware abstract layer (HAL) and kernel layer.

[0112] The application layer can include a series of application packages.

[0113] like Figure 4B As shown, the application package may include applications such as camera, gallery, calendar, map, music, navigation, SMS, call, video, voice control module, and one-handed mode control module.

[0114] The voice control module is used to respond to the voice command spoken by the user and execute the corresponding operation. The one-handed mode control module is used to respond to the one-handed mode trigger event, enter the small-screen display state of the one-handed mode, and display the small-screen display picture.

[0115] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0116] like Figure 4B As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and a multi-modal input framework, etc.

[0117] The window manager provides window management services (WMS). WMS can be used for window management, window animation management, surface management, and as a transfer station for the input system.

[0118] Content providers are used to store and retrieve data and make it accessible to applications. This data can include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0119] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0120] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0121] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0122] The activity manager can provide activity management services (AMS), which can be used to start, switch, and schedule system components (such as activities, services, content providers, and broadcast receivers) as well as manage and schedule application processes.

[0123] The input manager can provide input management services (IMS), which can be used to manage system input, such as touch screen input, key input, sensor input, etc. IMS takes events from input device nodes and distributes them to appropriate windows through interaction with WMS.

[0124] The multi-mode input framework encapsulates various interfaces for other subsystems and applications to call. In the embodiment of the present application, after determining the operation to be performed by the voice command based on the voice control function, the function module corresponding to the voice control function can send the instruction to the multi-mode input framework. Afterwards, the multi-mode input framework can perform the corresponding operation in response to the instruction, thereby achieving the effect of voice controlling the electronic device to perform the operation.

[0125] The Android runtime includes the core library and the Android runtime. The Android runtime is responsible for converting source code into machine code. The Android runtime mainly uses the ahead of time (AOT) compilation technology and the just in time (JIT) compilation technology.

[0126] The core library is mainly used to provide basic Java class library functions, such as basic data structures, mathematics, IO, tools, databases, networks, etc. The core library provides an API for users to develop Android applications.

[0127] The native C / C++ library can include multiple functional modules, such as surface manager, media framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0128] Among them, the surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media framework supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides drawing and operation of 2D graphics and 3D graphics in applications. SQLite provides a lightweight relational database for applications of electronic device 100.

[0129] The hardware abstraction layer runs in user space, encapsulates kernel layer drivers, and provides a calling interface to the upper layer.

[0130] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0131] The following is a brief introduction to the technical terms that may be involved in the embodiments of the present application.

[0132] Automatic Speech Recognition (ASR) technology aims to convert the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes, or character sequences.

[0133] Natural Language Understanding (NLU) is a general term for all methods, models or tasks that support machines to understand text content.

[0134] Dialogue Management (DM) is used to control the process of human-computer dialogue and decide the response to the user at the moment based on the dialogue history information.

[0135] The voice control method provided in the embodiment of the present application is mainly applied to electronic devices. Specifically, it can be applied to the scenario where the user controls the electronic device by voice in the small screen display state in the one-handed mode. For the simulated click voice command spoken by the user in the above scenario, the simulated click can be performed at the exact position.

[0136] Taking the electronic device as a mobile phone as an example, the voice control method provided in the embodiment of the present application may specifically include: after entering the small-screen display screen and starting the voice control function, the mobile phone collects audio data. The mobile phone performs voice recognition on the audio data. The mobile phone determines whether the operation corresponding to the audio data belongs to a simulated click operation. If so, the mobile phone obtains the simulated click position corresponding to the voice instruction of the simulated click on the small-screen display screen. Then, the mobile phone performs a simulated click operation according to this simulated click position. Specifically, in the case where the simulated click operation is an element click operation (such as the above-mentioned "open [calendar]"), the task triggered by the interface element corresponding to the element click operation is executed (such as opening the [calendar] application and displaying the calendar interface). When the simulated click operation is a sliding operation (such as the above-mentioned "slide down"), the simulated click start position and the simulated click end position are obtained, and then the display screen is slid from the simulated click start position to the simulated end position.

[0137] In the voice control method provided in the embodiment of the present application, it can be ensured that in the scenario where the user uses voice to control the electronic device to perform a simulated click operation in the one-handed mode, the mobile phone can perform the simulated click operation at the correct position to avoid the problem of simulated click errors. On the premise that the mobile phone accurately recognizes the user's voice command, the mobile phone can accurately perform the simulated click operation according to the user's voice command. Thereby, the problem of the click operation performed by the mobile phone being inconsistent with the intention of the voice command spoken by the user can be avoided.

[0138] Furthermore, the mobile phone first maps the first coordinate of the simulated click position in the small screen display obtained from the audio data in the screen coordinate system to the second coordinate of the corresponding position in the full screen display. Among them, the first mapping process of mapping the first coordinate to the second coordinate is specifically to map the point from the full screen display size of the full screen display to the small screen display size of the small screen display. Then, a simulated click event is generated based on the second coordinate, and the simulated click event is sent to the multi-mode input framework of the mobile phone. So that the multi-mode input framework responds to the simulated click event and performs a second mapping operation on the second coordinate to map it to the full screen display size.

[0139] Since the first coordinate in the small-screen display has been mapped to the first coordinate before, the second coordinate is obtained. When the multi-mode input framework performs a simulated click operation, the second coordinate is automatically assumed to be a point in the small-screen display. Afterwards, the second coordinate will be automatically mapped to the full-screen display, that is, the multi-mode input framework will perform a second mapping process on the second coordinate. And because the first mapping process is to map the full-screen display size to the small-screen display size, and the second mapping process is to map the small-screen display size to the full-screen display size. Then, based on the multi-mode input framework, the second mapping process is performed on the second coordinate to obtain the first coordinate. Finally, based on the multi-mode input framework, a simulated click operation is performed on the first coordinate to ensure that the simulated click operation is still the coordinate of the simulated click position in the audio data in the small-screen display. In other words, it ensures that the mobile phone can accurately perform the corresponding simulated click operation at the correct position according to the audio data.

[0140] The voice control method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0141] Please refer to Figure 5A , is a flow chart of a voice control method provided in some embodiments of the present application. In this embodiment, the method includes S500-S513:

[0142] S500. The phone starts.

[0143] S501. When the one-handed operation function is turned on, in response to a first event, a small-screen display image corresponding to the one-handed mode is displayed.

[0144] The first event may be a one-handed mode triggering event. Figure 3A After the switch 302 is shown, a preset gesture is performed on the mobile phone to trigger the mobile phone to enter the one-handed mode. In this embodiment, the first event may be specifically detecting a preset gesture. Exemplarily, the preset gesture may be sliding horizontally along the bottom of the screen and pausing, swiping downward from the bottom of the screen, sliding into the screen from the left or right corner of the bottom of the screen, and so on.

[0145] It is understandable that in other embodiments, if the user does not enable the one-handed operation function in advance, the mobile phone will not trigger the one-handed mode after detecting the preset gesture.

[0146] As can be seen from the above description, after entering the one-handed mode, the display image on the mobile phone screen will be reduced. Therefore, after detecting the first event, the mobile phone responds to the first event and changes the display image from the full-screen display size to the small-screen display size, such as Figure 3B As shown, in response to the first event, the mobile phone switches from display interface 303 to display interface 304 , or switches to display interface 306 .

[0147] S502. In response to the second event, start the voice control function.

[0148] In some embodiments, the second event may specifically correspond to a user manually turning on the voice control function. Figure 2A The switch 107 in the interface 105 shown in the figure is used to turn on the voice control function. In other embodiments, the second event may also correspond to the user turning on the intelligent human-computer interaction function. In this embodiment, after starting the intelligent human-computer interaction function, in order to better provide the user with an interactive method for the mobile phone, the mobile phone will automatically turn on the first voice control function. In other embodiments, the second event may also be other events.

[0149] S503. Collect audio data based on the voice control function.

[0150] As can be seen from the above description, in some embodiments, after the voice control function is activated, the mobile phone is in a state of continuously collecting audio data. In this embodiment, after the mobile phone activates the voice control function in S502, the mobile phone can collect audio data based on the voice control function. In some embodiments, the above S503 may specifically include: after the voice control function is activated, the mobile phone calls the recording device through the voice control module corresponding to the voice control function to start collecting audio data.

[0151] Furthermore, in some embodiments, after the voice control module calls the recording device to start collecting audio data, if the audio data is collected, the voice recognition engine corresponding to the voice control module needs to perform voice recognition on it to determine whether it contains voice instructions supported by the voice control function. Therefore, in some embodiments, after the voice control module calls the recording device, the above method also includes: starting the voice recognition engine corresponding to the voice control module. It should be noted that the specific implementation process of starting the voice recognition engine can refer to the description in the relevant technology, and will not be repeated in the embodiments of this application.

[0152] In other embodiments, the voice control function may also be a voice control function such as a free command. After the free command voice control function is started, it will trigger the collection of audio data and voice recognition only when a preset scene is triggered. In this embodiment, the above S503 may specifically include: after detecting that the mobile phone triggers a preset scene, collecting audio data based on the voice control function.

[0153] Alternatively, in other embodiments, the voice control function may also be a voice control function such as a voice assistant. After the mobile phone starts the voice assistant function, the voice assistant function may be awakened to start collecting audio data and perform voice recognition when a preset wake-up word is detected. In this embodiment, the above S503 may specifically include: after detecting the preset wake-up word, collecting audio data based on the voice control function.

[0154] It should be noted that the above-mentioned method of triggering the voice control function to start collecting audio data and performing voice recognition is only an example. In other embodiments, the voice control function can also trigger the mobile phone to start collecting audio data based on the voice control function in other ways.

[0155] S504. Perform voice recognition on the audio data to determine whether the audio data contains a preset voice instruction.

[0156] It can be seen from the above description that the voice control function can support multiple voice commands. Among them, the preset voice command can be any voice command among the voice commands supported by the voice control function. Combined with the above description, it can be seen that in some embodiments, the preset voice command can be a system command, such as: return to the desktop, return, open the application, slide up, slide down, slide left, slide right, brightness adjustment or volume adjustment, etc. The preset voice command can also be an application command, such as: play, pause, full screen, like, favorite, next, previous, fast forward or rewind, etc.

[0157] Alternatively, in other embodiments, the preset voice commands may also include different voice expressions corresponding to different commands, such as different voice expressions for different commands: desktop, return to desktop, return, previous page, open [application name], slide up, slide up, slide down, slide down, etc.

[0158] If the judgment result of S504 is no, that is, the mobile phone does not recognize that the audio data includes any voice command supported by the voice control function. At this time, it means that the user may not intend to control the mobile phone by voice. After that, the mobile phone can continue to collect audio data and perform voice recognition on the audio data to determine whether the user has spoken a voice command supported by the voice control function. That is, if the judgment result of S504 is no, the mobile phone can return to execute S503 to continue collecting audio data.

[0159] Alternatively, if the judgment result of S504 is no, it is also possible that the mobile phone has an error in recognizing the audio data and has failed to accurately recognize the voice command spoken by the user. In this case, the user needs to speak the accurate voice command again. In some embodiments, if the judgment result of S504 is no, the mobile phone can display a prompt message on the interface, which is used to prompt the user to try to speak the voice command again.

[0160] If the judgment result of S504 is yes, it means that the user wants to control the mobile phone to perform the corresponding operation by voice. Therefore, when the judgment result of S504 is yes, the mobile phone can perform the corresponding operation in response to the preset voice command.

[0161] S505: Determine whether the operation corresponding to the preset voice command is a simulated click operation.

[0162] As can be seen from the above description, in some embodiments, the simulated click operation may specifically include an element click operation and a sliding operation. Figure 2A The voice control function shown supports voice commands, and the voice commands corresponding to the element click operation may include application commands corresponding to opening the application, and the voice commands corresponding to the sliding operation may include system commands corresponding to swiping up, swiping down, swiping left, and swiping right.

[0163] In some embodiments, the mobile phone can simulate click operations for the above application instructions such as play, pause, full screen, like, favorite, previous, next, fast forward and rewind when executing voice control. In other embodiments, the mobile phone can also interact with the server of the corresponding application when executing the above application instructions through voice control.

[0164] Therefore, in some embodiments, the preset voice instructions corresponding to the simulated click operation may include any of the following: instructions corresponding to operations such as opening an application, swiping up, swiping down, swiping left, and swiping right in system instructions, and instructions corresponding to operations such as playing, pausing, full screen, liking, collecting, previous, next, fast forward, and rewinding in application instructions. Further, the above-mentioned simulated click operations can be divided into two categories: element click operations and sliding operations. Among them, element click operations can specifically include: opening an application, playing, pausing, full screen, liking, collecting, previous, next, fast forward, and rewinding. Sliding operations can include swiping up, swiping down, swiping left, and swiping right.

[0165] If the determination result of S505 is yes, it means that the user wishes to control the mobile phone to perform a simulated click operation through the voice control function.

[0166] S506: Obtain the simulated click position of the simulated click operation in the small-screen display image.

[0167] In some embodiments, the above S506 may specifically include: obtaining the first coordinate of the simulated click position in the small-screen display screen in the screen coordinate system. Taking the simulated click operation as an element click operation of "Open [Calendar]" as an example, the corresponding simulated click position in the small-screen display screen may be the position of the [Calendar] application icon in the small-screen display screen. And the simulated click position has a unique coordinate in the screen coordinate system, namely the above-mentioned first coordinate. Taking the simulated click operation as a sliding operation as an example, the corresponding simulated click position in the small-screen display screen may specifically include a simulated click start position and a simulated click end position in the small-screen display screen. The simulated click start position and the simulated click end position also correspond to a unique coordinate in the screen coordinate system respectively.

[0168] In some embodiments, when the simulated click operation is an element click operation, the mobile phone can first analyze all elements contained in the currently displayed screen and their corresponding position information. When it is determined that the operation corresponding to the audio data is an element click operation, the preset element corresponding to the element click operation is analyzed. Then, the position of the preset element is searched from the position information of all elements, which is the simulated click position corresponding to the element click operation in the small-screen display screen. In this way, it can be ensured that the simulated click position is the position in the small-screen display screen.

[0169] It can be seen from the above description that the mobile phone can pre-acquire the text data displayed in the current page, and analyze the text data to obtain all the words contained in the current page, as well as the corresponding positions of each word and its corresponding icon in the current page. Exemplarily, taking S504 and S505 as an example to determine that the preset voice command is "Open [Calendar]", and the words in the current page include the word [Calendar], then the mobile phone can obtain the position corresponding to the word [Calendar] in the current page. It can be understood that the simulated click position corresponding to the voice command "Open [Calendar]" is the position corresponding to the word [Calendar] in the current page. Taking the example that the position corresponding to the word [Calendar] in the current page includes the position of the application icon of the [Calendar] application, in this embodiment, in the above S506, the simulated click position obtained by the mobile phone can specifically be the position of the application icon of the [Calendar] application on the current page. Exemplarily, when the mobile phone is in the small-screen display state of the one-handed mode, the position corresponding to the word [Calendar] is Figure 3C Area 307 is shown.

[0170] In other embodiments, when the simulated click operation is an element click operation, the mobile phone may first determine the preset element corresponding to the element click operation. Then, the location of the preset element is searched from the currently displayed page as the simulated click location of the element click operation in the small-screen display. In this way, it can be ensured that the simulated click location is the location in the small-screen display.

[0171] It can be seen from the description of the above embodiment that the preset voice command corresponding to the simulated click operation, in addition to "open [application]", can also be a sliding command, such as swipe up, swipe down, swipe left, and swipe right. In this embodiment, the simulated click position may include two positions: the simulated click start position and the simulated click end position. Taking S504 and S505 as an example of determining that the preset voice command is "swipe down", in S506, the simulated click position corresponding to the simulated click operation obtained by the mobile phone may specifically include the following: Figure 3D D5 and D6 shown.

[0172] In other embodiments, when the audio data contains other instructions, S506 may also obtain the simulated click position corresponding to the simulated click operation in other ways.

[0173] S507. Obtain a full-screen display size in a full-screen display state, a small-screen display state in a one-handed mode, and a first mapping relationship from a full-screen display size to a small-screen display size.

[0174] In some embodiments, the mobile phone can obtain the full-screen display size of the mobile phone in the full-screen display state. After the mobile phone enters the small-screen display state of the one-handed mode, the small-screen display size of the small-screen display picture can be obtained. Then, according to the full-screen display size and the small-screen display size, a first mapping relationship between the mobile phone's transformation from the full-screen display size to the small-screen display size can be determined.

[0175] The full-screen display size may specifically include a first width and a second height, and the small-screen display size may include a second width and a second height. Furthermore, based on the above-mentioned first width and second height, the second width and second height, the width reduction ratio of the first width to the second width, and the height reduction ratio of the first height to the second height can be determined. Afterwards, combining the coordinates 1 in the full-screen display picture and the coordinates 2 in the small-screen display picture of the same interface element, as well as the width reduction ratio and the height reduction ratio, the first mapping relationship from the full-screen display size to the small-screen display size can be determined by coordinate conversion. In some embodiments, the first mapping relationship from the full-screen display size to the small-screen display size can actually also be expressed as a mapping relationship from coordinate 1 to coordinate 2.

[0176] From the above description, it can be seen that in different embodiments, the small screen display size of the small screen display state when the mobile phone enters the one-handed mode can be adjusted according to the needs of the user. In other words, in different embodiments, the small screen display size of the small screen display picture displayed by the mobile phone may be different.

[0177] S508. Based on the first mapping relationship, perform a first mapping process on the first coordinate corresponding to the simulated click position to obtain a second coordinate.

[0178] The first mapping process is used to map the coordinates of the point from the full-screen display image to the small-screen display image. Figure 3C The region 308 shown is mapped to the region 307. Specifically, each coordinate included in the region 308 can be mapped to obtain each coordinate of the region 307, or the coordinate of the center position of the region 308 can be mapped to obtain the coordinate of the center position of the region 307. Alternatively, the first mapping process is used to Figure 3D Position D5 in is mapped to position D3, and position D6 is mapped to D4.

[0179] From the above description, it can be seen that the first coordinate is the coordinate of the simulated click position in the small-screen display, and the first coordinate uses the screen coordinate system as the reference coordinate system. Then, the first mapping process is performed on the first coordinate, and the second coordinate obtained is still a coordinate using the screen coordinate system as the reference coordinate system. However, the second coordinate has no practical meaning in either the full-screen display or the small-screen display. Taking the preset voice command of "open [application]" as an example, the first coordinate of the simulated click operation obtained in the small-screen display is the corresponding position of the [Calendar] application in the small-screen display, that is, Figure 5B The position corresponding to the area 521 in the reduced main screen interface 520 is shown.

[0180] In addition, as can be seen from the above description, in one-handed mode, before the mobile phone performs a simulated click operation, the system (such as the multi-mode input framework) automatically sends the simulated click position (such as Figure 3C The area 307 is shown, and Figure 3D D3 and D4 shown in the figure) are automatically converted to the position in the full-screen display image (such as Figure 3C The area 308 is shown, and Figure 3D Then, a simulated click operation is performed according to the position in the full screen display. Figure 3C and Figure 3D As shown, in the one-handed mode, the simulated click operation is performed according to the position in the full-screen display, which is the wrong position. The position where the simulated click should actually be performed should be the position in the small-screen display, that is, the first coordinate.

[0181] Therefore, in order to ensure that the position of the simulated click performed by the mobile phone is the position in the small screen display, the simulated click position needs to be mapped from the first coordinate to the virtual second coordinate. Figure 5B The area 521 shown in FIG. 1 is subjected to a first mapping process (eg Figure 5B L1 mapping process shown in the figure) to obtain a virtual area 522. Figure 5B It can be seen that the second coordinate obtained by performing the first mapping process on the first coordinate does not correspond to the position of the [Calendar] application in the full-screen display screen or the small-screen display screen. In this way, when the system performs a simulated click operation, the first coordinate can be obtained when the second coordinate is automatically mapped (from the small-screen display screen to the full-screen display screen). Finally, the mobile phone can perform a simulated click operation on the position where the first coordinate is located.

[0182] S509. Generate a simulated click event based on the second coordinate.

[0183] S510. Send a simulated click event to the multi-mode input framework of the mobile phone.

[0184] From the above description, it can be seen that in the one-handed mode, when the mobile phone system performs a simulated click operation, the coordinates (second coordinates) obtained from the simulated click event are regarded as coordinates in the small screen display image, and the coordinates are automatically processed by the second mapping to the coordinates in the full screen display image. Finally, the simulated click operation is performed on the position corresponding to the coordinates in the full screen display image obtained after the second mapping.

[0185] S511. Based on the multi-modal input framework, perform a second mapping process on the second coordinate to obtain the first coordinate.

[0186] Among them, the second mapping process is used to map the coordinates of the point from the small-screen display to the full-screen display. It can be understood that the second mapping process and the first mapping process are two opposite mapping processes. Exemplarily, taking the mapping process from point A to point C as the first mapping process, the second mapping process is the mapping process from point C to point A. Therefore, when the mobile phone system performs the second mapping process on the second coordinate and converts it to the coordinate in the full-screen display, the first coordinate is obtained. Please refer to Figure 5B As shown, the above S511 performs a second mapping process on the area 522 corresponding to the second coordinate (such as Figure 5B The L2 mapping process shown in FIG. 5 ) can obtain the area 521 corresponding to the first coordinate.

[0187] For example, if the phone enters the right-handed mode in the one-handed mode, please refer to Figure 5C, construct the mobile phone screen coordinate system XOY and the window coordinate system X`O`Y`. The scaling ratio of the small screen display size to the full screen display size on the X axis is recorded as K x , the scaling factor on the Y axis is denoted as K y It should be noted that the screen coordinate system is the coordinate system corresponding to the full-screen display screen when the mobile phone is in full-screen display state, and the window coordinate system is the coordinate system corresponding to the small-screen display screen when the mobile phone is in small-screen display state.

[0188] C(X0, Y0) represents the simulated click position obtained by the mobile phone, which is the position in the small screen display size. The coordinates of point C still use the origin O of the screen coordinate system as the reference point. For example, point C corresponds to the first coordinates above, such as Figure 3C The area 307 is shown. In some embodiments, the coordinates of point C are absolute coordinates relative to the screen coordinate system, that is, the values ​​of X0 and Y0 are described with the origin O of the screen coordinate system as the reference point.

[0189] A(X1, Y1) represents the position corresponding to the simulated click position C after being mapped to the full screen display size. For example, point C corresponds to Figure 3C Area 308 is shown.

[0190] As can be seen from the figure, the intersection of the first auxiliary line passing through point A and perpendicular to the X-axis and the second auxiliary line passing through point C and perpendicular to the Y-axis is B(X1,Y0). C`(X`,Y`) is the position of the lower right corner of the screen. B`(X1,Y`) is the projection point of point B on DC`.

[0191] From the above description, we can know that the aspect ratio of the full screen display size of the mobile phone is the same as that of the small screen display size. In other words, the reduction ratio of the width of the small screen display size to the width of the full screen display size is the same as the reduction ratio of the height. By similar principle:

[0192]

[0193]

[0194] Right now,

[0195]

[0196]

[0197] From the above description, it can be seen that in the small screen display state of the one-handed mode, when the mobile phone system performs the simulated click operation, it will automatically convert the simulated click position from the position in the small screen display state to the position in the full screen display state. That is, if the simulated click position is C, the mobile phone system will automatically convert point C to point A and perform the simulated click operation at point A. It can be understood that point A is an incorrect simulated click position.

[0198] It is understandable that the conversion process performed by the mobile phone system when automatically converting the simulated click position from point C in the small screen display state to point A in the full screen display state can be expressed by the following two formulas. The actual click position A (X1, Y1) changes based on C (X0, Y0):

[0199]

[0200]

[0201] In order to ensure that the final simulated click position is the accurate position, i.e., the above-mentioned point C, when the mobile phone performs a simulated click operation in the one-handed mode, point Z can be obtained by first converting point C. Thus, when the mobile phone system performs a simulated click operation, point Z is converted to point C in the process of automatically mapping the input coordinates, and finally the simulated click operation can be performed at point C. The coordinates of point Z are marked as (x, y).

[0202] From the above description, it can be seen that formula (1-5) represents the change process of the coordinates on the X axis in the automatic conversion step when the mobile phone system performs a simulated click operation, and formula (1-6) represents the change process of the coordinates on the Y axis in the automatic conversion step when the mobile phone system performs a simulated click operation. Therefore, in order to enable the mobile phone system to convert point Z (x, y) to point C (X0, Y0) when performing the automatic conversion step, X0 = x, X1 = X0 in the above formula (1-5) can be used, so that the value of x can be calculated:

[0203] x=K x (X0-X')+X' (1-7)

[0204] Similarly, in formula (1-6), let Y0 = y, Y1 = Y0, and calculate the value of y:

[0205] y=K y (Y0-Y')+Y' (1-8)

[0206] It can be understood that the above formulas (1-7) and (1-8) correspond to the above-mentioned first mapping process. That is to say, if the mobile phone generates a simulated click event with Z(x, y), the system can obtain the coordinates of the above-mentioned simulated click position in the small screen display after performing the automatic conversion step, that is, point C(X0, Y0). Among them, the values ​​of x and y can be calculated according to C(X0, Y0) through formulas (1-7) and (1-8). Exemplarily, Z(x, y) is the above-mentioned second coordinate.

[0207] It should be noted that the above Figure 5B and Figure 5C The coordinate mapping process shown in the demonstration example is only an example of a reduced scale right-handed single-handed mode. In other embodiments, the mobile phone displays other single-handed mode display screens (such as left-handed single-handed mode (such as Figure 3B When the reduced main screen interface 306 is displayed, or a half-screen display screen is displayed, etc.), the conversion process of the first mapping process and the second mapping process can be other formulas.

[0208] S512. Based on the multi-modal input framework, perform a simulated click operation at the position corresponding to the first coordinate.

[0209] Taking the simulated click operation as an element click operation as an example, the above S512 may specifically include: executing a task triggered by an interface element corresponding to the element click operation based on the multi-modal input framework. For example, for the voice command "open [calendar]", the mobile phone will control the simulated click at the [calendar] application icon, so that the mobile phone opens the [calendar] application.

[0210] Taking the simulated click operation as a sliding operation as an example, the above S512 may specifically include sliding the display screen from the simulated click start position to the simulated click end position based on the multi-modal input framework.

[0211] It should be noted that the specific implementation process of the mobile phone performing a simulated click operation at a certain position based on the multi-mode input framework can refer to the description in the relevant technology and will not be repeated in the embodiments of the present application.

[0212] In some embodiments, while executing the simulated click operation in response to the voice command, the mobile phone can also indicate where the simulated click operation was executed by displaying the simulated click track in the display screen. In this way, the user can easily view the execution location of the simulated click, making the process of voice-controlled mobile phone simulated click more intuitive. Figure 6In the small screen display state, the mobile phone displays a reduced home screen display interface 620. In the reduced home screen display interface 620, the user speaks a voice command "Open [Calendar]". In response to the voice command, the mobile phone displays a simulated click track 621 at the position corresponding to the word [Calendar]. And at the position corresponding to the simulated click position track 621, a simulated click operation is performed to open the calendar application and enter the following Figure 6 The reduced calendar application interface 622 is shown. Similarly, when performing a sliding operation in response to a voice instruction of a sliding operation spoken by the user, the mobile phone can also display simulated click tracks in sequence along the track of the simulated sliding operation, or display simulated click tracks at the starting point and the end point of the simulated sliding operation respectively.

[0213] In addition, it should be noted that in some other embodiments, if the judgment result of the above S505 is no, it means that the user wants to control the operation performed by the mobile phone through the voice control function, which does not belong to the simulated click operation. At this time, in response to this voice command, the mobile phone can directly perform the corresponding operation. For example, after determining that the audio data includes the preset voice command "return to the desktop", in response to the voice command, the mobile phone can directly control the display interface of the mobile phone to return to the desktop, such as Figure 5A S513 shown.

[0214] S513. In response to the preset voice command, the mobile phone performs a corresponding operation.

[0215] In the technical solution provided by the embodiment of the present application, in the one-handed mode, if the voice command spoken by the user is not a simulated click command, then when the mobile phone performs the corresponding operation, it is the same as when the mobile phone is in full-screen display state and receives the voice control execution command. That is to say, at this time, the mobile phone responds to the voice command and normally performs the corresponding operation, and can also accurately realize the user's intention to control the mobile phone.

[0216] In the technical solution provided in the embodiment of the present application, when the simulated click operation is an element click operation (such as the above-mentioned "open [calendar]"), the task triggered by the interface element corresponding to the element click operation is executed (such as opening the [calendar] application and displaying the calendar interface). When the simulated click operation is a sliding operation (such as the above-mentioned "slide down"), the simulated click starting position and the simulated click ending position are obtained, and then the display screen is slid from the simulated click starting position to the simulated end position. In this way, it can be ensured that in the one-handed mode, in the scenario where the user uses voice to control the electronic device to perform a simulated click operation, the mobile phone responds to the voice instruction of the simulated click, can perform the simulated click operation at the correct position, trigger the corresponding response operation, and avoid the problem of simulated click errors. On the premise that the mobile phone accurately recognizes the user's voice instructions, the mobile phone can respond to the voice instruction of the simulated click and accurately perform the simulated click operation.

[0217] It can be seen from the description of the above embodiments that the embodiments of the present application are mainly aimed at the scenario in which the user controls the mobile phone through voice to perform a simulated click operation in one-handed mode. Specifically, after determining that it is in the above scenario, before executing the voice instruction of the simulated click, the determined simulated click position is first mapped, and then the simulated click operation is performed. In order to ensure that the mobile phone can accurately identify the one-handed mode scenario before performing the simulated click operation, the mobile phone can determine whether the current mobile phone is in the small-screen display state in the one-handed mode after starting the voice control function and receiving the audio data. If so, when voice recognition is performed on the audio data and it is determined that it contains a simulated click voice instruction supported by the voice control function, the simulated click position corresponding to the voice instruction of the simulated click is first mapped, and then the simulated click operation is performed.

[0218] In other embodiments, in order to ensure that the mobile phone can accurately identify the one-handed mode scenario before executing the simulated click operation, the mobile phone can also perform voice recognition on the audio data, and if it is determined that the audio data contains the voice command of the simulated click supported by the voice control function, it is determined whether the current mobile phone is in the small screen display state of the one-handed mode. If so, the simulated click position is mapped first, and then the simulated click operation is executed.

[0219] Exemplarily, in some embodiments, before S507, the method may further include determining whether the mobile phone is in a small screen display state in a one-handed mode. In this embodiment, when the mobile phone is currently in a small screen display state, S507-S512 may be executed.

[0220] In the technical solution provided in the embodiment of the present application, after each time the mobile phone receives audio data based on the voice control function, before executing the simulated click operation in response to the voice instruction of the simulated click contained in the audio data, it will first determine whether the current mobile phone is in the small-screen display state of the one-handed mode. If it is determined that the current mobile phone is in the small-screen display state of the one-handed mode, for the simulated click operation to be executed, it is necessary to first perform coordinate mapping, and then perform the simulated click operation according to the coordinates obtained after mapping. In this way, it can be ensured that the mobile phone can perform the simulated click operation at the exact position in response to the voice instruction in the small-screen display state of the one-handed mode. Thereby accurately performing the corresponding simulated click operation according to the voice control instruction of the user to perform the simulated click.

[0221] In both of the above cases, it is determined whether the current mobile phone is in the small-screen display state of the one-handed mode after receiving the audio data. In other words, each time the mobile phone collects audio data based on the voice control function, it needs to determine whether the audio data is in the small-screen display state of the one-handed mode. Therefore, in order to reduce the number of times the voice control function determines whether it is in the small-screen display state of the one-handed mode, in other embodiments, the mobile phone can also send a first notification message to the voice control module corresponding to the voice control function after detecting the small-screen display state entering the one-handed mode. The first notification message is used to inform the voice control module that it has entered the small-screen display state of the one-handed mode. In this way, when the voice control function receives and recognizes a voice instruction containing a simulated click, the simulated click position can be mapped first, and then the simulated click operation can be performed.

[0222] It should be noted that the mobile phone can turn on the voice control function before entering the small-screen display state of the one-handed mode. In this way, the mobile phone detects that it has entered the small-screen display state of the one-handed mode and immediately sends a first notification message to the voice control module.

[0223] In other embodiments, the mobile phone may also turn on the voice control function after entering the small screen display state of the one-handed mode. In this embodiment, when entering the small screen display state of the one-handed mode, since the voice control function is not turned on, the first notification message cannot be sent to the voice control module immediately. At this time, it is possible to monitor whether the voice control function is turned on. Afterwards, after detecting that the voice control function is turned on by the mobile phone, the first notification message is sent to the voice control module.

[0224] In the technical solution provided by the embodiment of the present application, the voice control module corresponding to the voice control function does not need to judge whether the mobile phone has entered the small screen display state of the one-handed mode every time it receives audio data. This can reduce the number of times the voice control module judges the one-handed mode and reduce power consumption.

[0225] Furthermore, in an embodiment in which the mobile phone sends a first notification message to the voice control module after entering the small-screen display state of the one-handed mode, the mobile phone should send a second notification message to the voice control module after detecting the small-screen display state of exiting the one-handed mode. The second notification message is used to inform the voice control module that the current mobile phone has exited the small-screen display state of the one-handed mode. It can be understood that after receiving the second notification message, the voice control module no longer maps the received audio data containing the simulated click instruction, but can directly perform the simulated click operation according to the obtained simulated click position.

[0226] In the technical solution provided in the embodiment of the present application, after detecting that the mobile phone has exited the small-screen display state of the one-handed mode, if a preset voice instruction containing a simulated click operation is received based on the voice control function, it is no longer necessary to perform the first mapping process on the acquired simulated click position. In this way, after the mobile phone enters the full-screen display state, the mobile phone can also perform a simulated click operation at an accurate position for the voice instruction uttered by the user that requires a simulated click operation.

[0227] Please refer to Figure 7 , which is a schematic diagram of the interaction between the internal modules of a mobile phone during the execution of the voice control method provided in an embodiment of the present application. In this embodiment, the mobile phone includes: a voice control module, a content sensor (ContentSensor), a view grabber (ViewFetcher), a speech recognition engine (specifically, ASR / DM / NLU), an instruction execution module (ActionExecutor), a system settings module (Settings), a window manager (WindowManager), a one-handed mode control module, and an event manager (EventManagerEx). In some embodiments, the above-mentioned event manager belongs to a multi-modal input framework. In this embodiment, the voice control method may specifically include:

[0228] After the voice control function is started, the voice control module uses the cross-process call framework layer interface to make the content sensor call the top-level activity result callback function. This allows the view grabber to obtain the text data in the current page, analyze the text data, and obtain the words contained in the current page. After that, the view grabber returns the words to the voice control module to return the words contained in the current page.

[0229] The voice control module sends the words to the voice recognition engine. And after acquiring the audio data, the audio data is sent to the voice recognition engine.

[0230] The voice recognition engine determines whether the audio data contains a preset voice instruction and whether the operation corresponding to the preset voice instruction is a simulated click operation. In addition, if the voice recognition engine determines that the audio data contains a preset voice instruction for a simulated click operation, the instruction execution module is notified to execute the simulated click operation.

[0231] The instruction execution module registers a one-handed mode listener with the system setting module. Through the one-handed mode listener, it monitors whether the current state is in one-handed mode. If so, the one-handed mode control module obtains the full-screen display size of the mobile phone through the window management module. And obtains the zoom ratio of the one-handed mode from the system settings. The zoom ratio of the small-screen display size in the current small-screen display state to the full-screen display size. The one-handed mode control module calculates the second coordinate corresponding to the target position according to the first coordinate of the simulated click position corresponding to the preset voice command and the zoom ratio (such as Figure 5C The coordinates of the point Z shown).

[0232] Afterwards, a simulated click event is generated based on the second coordinate. Then, the instruction execution module is injected into the event manager to simulate the click event. After receiving the simulated click event, the event manager parses and obtains the second coordinate carried in the simulated click event. In response to the simulated click event, the second coordinate is mapped from the small-screen display size to the full-screen display size. That is, the second coordinate is mapped and converted to obtain the first coordinate of the simulated click position in the small-screen display size. Finally, the mobile phone can perform a simulated click operation on the location of the first coordinate.

[0233] In the technical solution provided in the embodiment of the present application, before executing the simulated click operation in response to the simulated click voice command, the simulated click position has been converted and mapped to the small screen display image displayed in the one-handed mode. Afterwards, the simulated click operation is executed according to the converted position coordinates, which can ensure that the position where the simulated click operation is executed is accurate. This can avoid the problem that the click operation executed by the mobile phone is inconsistent with the intention of the voice command spoken by the user.

[0234] Some other embodiments of the present application provide an electronic device (such as a mobile phone). The electronic device may include: a memory, a display screen, a recording device and one or more processors. The memory, the display screen and the recording device are coupled to the processors respectively. The display screen is used to display the interface of the electronic device, and the recording device is used to collect audio data. The memory is also used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various functions or steps executed by the mobile phone in the above method embodiment. The structure of the electronic device can refer to Figure 4A The structure of the electronic device 100 is shown.

[0235] The present application also provides a chip system, such as Figure 8As shown, the chip system 80 includes at least one processor 801 and at least one interface circuit 802. The processor 801 and the interface circuit 802 can be interconnected via lines. For example, the interface circuit 802 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 802 can be used to send signals to other devices (such as processor 801). Exemplarily, the interface circuit 802 can read instructions stored in the memory and send the instructions to the processor 801. When the instruction is executed by the processor 801, the electronic device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0236] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device (such as a mobile phone), the electronic device executes each function or step executed by the mobile phone in the above-mentioned method embodiment.

[0237] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the mobile phone in the above method embodiment. The computer may be an electronic device, such as a mobile phone.

[0238] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0239] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0240] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0241] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0242] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0243] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A voice control method, characterized in that: The method is applied to an electronic device having a voice control function and a one-hand operation function; the method comprises: In response to a one-handed mode triggering event, switching from a full-screen display image to a small-screen display image in a one-handed mode display state; After the voice control function is activated, voice recognition is performed on the audio data collected by the electronic device based on the voice control function to obtain a voice recognition result; In a case where the speech recognition result indicates that the operation corresponding to the audio data is a simulated click operation, determining a simulated click position corresponding to the simulated click operation in the small-screen display image; the simulated click operation includes an element click operation or a sliding operation; when the simulated click operation is the sliding operation, the simulated click position includes a simulated click start position and a simulated click end position; Execute a simulated click operation according to the simulated click position; If the simulated click operation is the element click operation, the electronic device executes the task triggered by the interface element corresponding to the element click operation; if the simulated click operation is the sliding operation, the electronic device slides the display screen from the simulated click starting position to the simulated click ending position.

2. The method according to claim 1, characterized in that: The determining a simulated click position corresponding to the simulated click operation in the small-screen display image includes: Determine a first coordinate of the simulated click position in the small screen display image in a screen coordinate system; Before performing the simulated click operation according to the simulated click position, the method further includes: Performing a first mapping process on the first coordinates to obtain a second coordinate in the screen coordinate system; the first mapping process is used to map the coordinates of the point from the full-screen display picture to the small-screen display picture; The performing a simulated click operation according to the simulated click position includes: generating a simulated click event based on the second coordinate; Sending the simulated click event to a multi-mode input framework of the electronic device; In response to the simulated click event, the multi-mode input framework performs a second mapping process on the second coordinate to obtain the first coordinate; the second mapping process is used to map the coordinate of the point from the small-screen display picture to the full-screen display picture; Based on the multi-modal input framework, a simulated click operation is performed on the position of the first coordinate.

3. The method according to claim 2, characterized in that The performing a first mapping process on the first coordinate to obtain a second coordinate in the screen coordinate system includes: Acquire a full-screen display size of the electronic device in the full-screen display picture, a small-screen display size in the small-screen display picture, and a first mapping relationship from the full-screen display size to the small-screen display size; Based on the first mapping relationship, a first mapping process is performed on the first coordinate to obtain the second coordinate.

4. The method according to claim 3, characterized in that The obtaining of the full-screen display size of the electronic device in the full-screen display picture, the small-screen display size in the small-screen display picture, and a first mapping relationship from the full-screen display size to the small-screen display size includes: Obtain a first width and a first height of the full-screen display size, and a second width and a second height of the small-screen display size; Determine, according to the first height and the second height, a height reduction ratio for converting the full-screen display size into the small-screen display size; Determining, according to the first width and the second width, a width reduction ratio for converting the full-screen display size into the small-screen display size; The first mapping relationship is determined according to the height reduction ratio and the width reduction ratio.

5. The method according to any one of claims 2 to 4, characterized in that: The performing a second mapping process on the second coordinate to obtain the first coordinate includes: Acquire a second mapping relationship for transforming the electronic device from the small-screen display size to the full-screen display size; Based on the second mapping relationship, a second mapping process is performed on the second coordinate to obtain the first coordinate.

6. The method according to any one of claims 2 to 5, characterized in that: The simulated click operation is the element click operation; and determining a simulated click position corresponding to the simulated click operation in the small-screen display image includes: Analyze the preset element corresponding to the element click operation; and search for the position corresponding to the preset element in the small-screen display image.

7. The method according to any one of claims 2 to 5, characterized in that: The simulated click operation is the element click operation; and determining a simulated click position corresponding to the simulated click operation in the small-screen display image includes: Obtaining all elements contained in the small-screen display image and position information corresponding to each element; Analyze the preset element corresponding to the element click operation; The position corresponding to the preset element is found according to the position information.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: While the simulated click operation is being performed, a simulated click track is displayed at a position corresponding to the simulated click operation.

9. The method according to any one of claims 1 to 8, characterized in that The aspect ratio of the full-screen display image is the same as the aspect ratio of the small-screen display image.

10. An electronic device, characterized in that: The electronic device comprises: a processor, a memory, a display screen and a recording device; the memory, the display screen and the recording device are respectively coupled to the processor; The display screen is used to display the interface of the electronic device, and the recording device is used to collect audio data; the memory stores computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 9.

Images