Display devices and voice control methods

Abstract

This application provides a display device and a voice control method. The display device includes a monitor, a sound acquisition unit, and a controller connected to both the monitor and the sound acquisition unit. The monitor is configured to display an image and a user interface. The sound acquisition unit is configured to acquire user voice control commands. The controller is configured to: determine at least one scrolling control in the current display interface; construct a list of voice scrolling control words for the scrolling controls in the current display interface; the voice scrolling control word list represents the correspondence between the scrolling direction of the scrolling control and the semantic control words; and, in response to the user's voice control command, control the target scrolling control in the current display interface to perform a scrolling operation based on the voice scrolling control word list and the corresponding voice control text. This achieves voice control of the scrolling control, improving the flexibility and convenience of the display device control method and enhancing the user experience.

Description

Technical Field

[0001] This application relates to the field of voice control technology, and in particular to a display device and a voice control method. Background Technology

[0002] With the development of voice control technology, voice functions have been integrated into various display devices (such as smart TVs), allowing users to jump to / open specific applications via voice commands.

[0003] Typically, each display device maintains multiple controls on its interface that respond to voice control commands. These controls can display corresponding interface descriptions. Upon receiving a user's voice control command, the display device matches the voice control text corresponding to the command with the interface descriptions of each control on the current display screen. If a match is found, the device controls the corresponding control to perform the relevant operation. For example, if a resource library control is displayed on the current screen, in response to the voice control command "Open resource library," the display device can automatically select the resource library control and open the resource library to display various multimedia resources for the user to choose from.

[0004] However, some display interfaces contain controls without descriptive text, such as scroll controls. Because scroll controls lack descriptive text and their scrolling direction is not unique, the aforementioned method of matching descriptive text for voice control is not applicable. Summary of the Invention

[0005] This application provides a display device and a voice control method that can effectively control scrolling controls on the display interface that do not have interface descriptive words.

[0006] In a first aspect, this application provides a display device, including a display, a sound acquisition unit, and a controller connected to the display and the sound acquisition unit respectively, wherein:

[0007] The monitor is configured to display images and a user interface.

[0008] The voice acquisition device is configured to collect the user's voice control commands;

[0009] The controller is configured as follows:

[0010] Identify at least one scroll control contained in the currently displayed interface;

[0011] Construct a list of voice scrolling control words for the scrolling controls in the current display interface; the list of voice scrolling control words is used to represent the correspondence between the scrolling direction of the scrolling controls and the semantic control words;

[0012] In response to the user's voice control command, based on the list of voice scrolling control words and the corresponding voice control text, the target scrolling control is controlled to perform a scrolling operation in the scrolling direction indicated by the voice control text; the target scrolling control is one of the scrolling controls in the currently displayed interface that corresponds to the voice control text.

[0013] In some embodiments, the controller is further configured to:

[0014] After the target application starts, monitor changes in the target application's display interface;

[0015] If the display interface changes, obtain the control information of multiple controls contained in the current display interface after the change;

[0016] Based on the control information, determine at least one scrolling control contained in the currently displayed interface.

[0017] In some embodiments, the controller is further configured to:

[0018] Get the display positions of multiple child controls controlled by the scroll control in the current display interface;

[0019] Determine the scrolling direction of the scroll control based on the display position of each sub-control;

[0020] Based on the scrolling direction, construct a list of voice scrolling control words for the scrolling controls in the currently displayed interface.

[0021] In some embodiments, the controller is further configured to:

[0022] If all child controls have the same display height in the current display interface, then the scrolling direction of the scrolling control is determined to be horizontal scrolling;

[0023] If all child controls have the same display width in the current display interface, then the scrolling direction of the scrolling control is determined to be vertical scrolling;

[0024] If the display height and / or display width of each child control are different in the current display interface, then the scrolling direction of the scrolling control is determined to be waterfall scrolling.

[0025] In some embodiments, the controller is further configured to:

[0026] Get the voice control text corresponding to the voice control command;

[0027] Match the voice control text with the semantic control words in the voice scrolling control word list to determine the target scrolling control that the voice control text requests to control in the current display interface, as well as the target scrolling direction of the target scrolling control;

[0028] Based on the voice control text, control the target scroll control to perform a scrolling operation in the target scrolling direction.

[0029] In some embodiments, the controller is further configured to:

[0030] Based on the voice control text, obtain the target scroll distance of the target scroll control;

[0031] Control the target scroll control to perform a scrolling operation in the target scrolling direction according to the target scrolling distance.

[0032] In some embodiments, the voice control text includes page-turning voice text, and the controller is further configured to:

[0033] If the target scrolling direction is horizontal, the target scrolling distance is calculated based on the display positions of the rightmost and leftmost child controls controlled by the target scrolling control.

[0034] If the target scrolling direction is vertical scrolling or waterfall scrolling, the target scrolling distance is calculated based on the display positions of the topmost and bottommost sub-controls controlled by the target scrolling control.

[0035] In some embodiments, if the current display interface contains multiple candidate scroll controls corresponding to the voice control text, the controller is further configured to:

[0036] Obtain the control priority of each candidate scroll control;

[0037] The target scroll control is determined from multiple candidate scroll controls based on control priority.

[0038] In some embodiments, the controller is further configured to:

[0039] Send the voice control command to the cloud server to request the cloud server to parse and process the voice control command;

[0040] Receive the voice control text corresponding to the voice control commands sent by the cloud server.

[0041] Secondly, this application provides a voice control method, including:

[0042] Identify at least one scroll control contained in the currently displayed interface;

[0043] Construct a list of voice scrolling control words for the scrolling controls in the current display interface; the list of voice scrolling control words is used to represent the correspondence between the scrolling direction of the scrolling controls and the semantic control words;

[0044] In response to the user's voice control command, based on the list of voice scrolling control words and the corresponding voice control text, the target scrolling control is controlled to perform a scrolling operation in the scrolling direction indicated by the voice control text; the target scrolling control is one of the scrolling controls in the currently displayed interface that corresponds to the voice control text.

[0045] Thirdly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a controller in a display device, can implement some or all of the steps of the voice control method provided in this application.

[0046] Fourthly, this application also provides a computer program product, which includes a computer program that, when executed by a controller in a display device, can implement some or all of the steps of the voice control method provided in this application.

[0047] The technical solution provided in this application can achieve at least the following beneficial effects:

[0048] In the display device and voice control method provided in this application, the controller in the display device is connected to both the display and the sound acquisition device. During the display of images and the user interface, the controller identifies at least one scrolling control in the current display interface and constructs a list of voice scrolling control words for the scrolling controls. This list represents the correspondence between the scrolling direction of the scrolling control and the semantic control words. Further, after detecting and acquiring a user's voice control command, the sound acquisition device sends the command to the controller. The controller responds to the command by controlling the target scrolling control corresponding to the voice control command in the current display interface to perform a scrolling operation based on the pre-constructed list of voice scrolling control words and the corresponding voice control text. Thus, by pre-identifying the scrolling control and constructing a list of voice scrolling control words for it, the display device can control the target scrolling control in the current display interface to perform a scrolling operation according to the user's voice control command when it receives the command. This overcomes the limitation that scrolling controls cannot be directly controlled by voice, achieving full voice control of all controls in the display device, improving the flexibility and convenience of the display device's control method, and enhancing the user experience. Attached Figure Description

[0049] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0050] Figure 1 This application illustrates an operational scenario between a display device and a control device, as shown in some embodiments.

[0051] Figure 2 This is a hardware configuration block diagram of a display device shown in some embodiments of this application;

[0052] Figure 3 This is a software configuration block diagram of a display device shown in some embodiments of this application;

[0053] Figure 4 This is a schematic diagram of the control distribution of a display interface shown in some embodiments of this application;

[0054] Figure 5 This is a schematic flowchart illustrating a voice control method performed by a display device, as shown in some embodiments of this application;

[0055] Figure 6 This is a schematic diagram of the horizontal scrolling of a scrolling control shown in some embodiments of this application;

[0056] Figure 7 This is a schematic diagram of vertical scrolling of a scrolling control shown in some embodiments of this application;

[0057] Figure 8 This is a schematic diagram of a waterfall scrolling control shown in some embodiments of this application;

[0058] Figure 9 This is a schematic diagram illustrating the process of a voice-controlled scrolling control performing a scrolling operation, as shown in some embodiments of this application;

[0059] Figure 10 This is a schematic diagram illustrating the distribution of multiple scrolling controls in a display interface, as shown in some embodiments of this application;

[0060] Figure 11 This is a schematic diagram illustrating the voice control logic of a scrolling control in a display interface, as shown in some embodiments of this application. Detailed Implementation

[0061] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of the technical solutions of this application, and not all embodiments.

[0062] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0063] The terms "comprising" and "having" and their variations mentioned in the specification and claims of this application are intended to cover but not exclude inclusion. For example, a product / device that includes a series of components is not necessarily limited to all the components that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such products / devices.

[0064] In the embodiments of this application, the display device can have various forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, or a personal computer, a laptop computer, a smartphone, a tablet computer, a portable wearable device, etc.

[0065] The aforementioned display devices can be controlled via control devices, intelligent control devices, voice, actions, gestures, trigger actions, etc.

[0066] See Figure 1 In an exemplary control operation scenario between a display device and a control device, a user can operate the display device 200 via the smart device 300 or the control device 100.

[0067] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0068] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the smart device may be used to control the display device 200.

[0069] In some embodiments, the display device 200 may receive user control via touch or gestures instead of using the smart device 300 or control device 100 to receive instructions.

[0070] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the smart device 300. For example, it can be controlled by directly receiving user voice commands through a module configured inside the display device 200 for acquiring voice commands; or it can be controlled by receiving user voice commands through a voice control device external to the display device 200.

[0071] In some embodiments, the display device 200 also communicates with the server 400, allowing the display device 200 to communicate via a local area network (LAN), wireless local area network (WLAN), or other networks.

[0072] The server 400 can provide various content and interactive features to the display device 200.

[0073] As an example, server 400 can be a standalone server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0074] See Figure 3 The display device 200 may include at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory 280, a power supply, and a user interface.

[0075] In some embodiments, the tuner 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals from multiple wireless or wired broadcast television signals. For example, Electronic Program Guide (EPG) data signals.

[0076] In some embodiments, the tuner 210 and the controller 250 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0077] In some embodiments, the communicator 220 is a component for communicating with external devices or the server 400 according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver.

[0078] In some embodiments, the display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 via the communicator 220.

[0079] For example, a display device can communicate with a cloud server through a communicator, sending data information that needs to be processed to the cloud server and receiving the processed data information from the cloud server.

[0080] In some embodiments, detector 230 is used to collect signals of the external environment or the interaction between display device 200 and the outside world. For example, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound acquisition device, such as a microphone, for collecting external sounds.

[0081] In the interactive scenario of the voice-controlled display device shown in this application, the sound collector can be used to collect the user's voice control commands.

[0082] In some embodiments, the external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), Composite Video Broadcast Signal (CVBS), Universal Serial Bus (USB) input interface, RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0083] In some embodiments, the controller 250 includes at least one of the following: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), a RAM (RandomAccess Memory), a ROM (Read-Only Memory), a first to an nth interface for input / output, a communication bus, etc.

[0084] The controller 250 can control the operation of the display device 200 and respond to user operations through various software control programs stored in the memory 280. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0085] In the interactive scenario of the voice-controlled display device shown in this application, the controller is configured to determine at least one scrolling control contained in the current display interface and construct a list of voice scrolling control words for the scrolling controls in the current display interface; then, after the sound acquisition unit receives the user's voice control command, based on the list of voice scrolling control words and the voice control text corresponding to the voice control command, the controller controls the target scrolling control corresponding to the voice control text to perform a scrolling operation in the scrolling direction indicated by the voice control text among the scrolling controls contained in the current display interface.

[0086] In some embodiments, the display 260 includes a display screen component for presenting an image and a driving component for driving the image display. The display 260 is used to receive image signals output by the controller 250 and to display video content, image content, menu control interface components, and user control UI interface.

[0087] As an example, the display 260 can be a liquid crystal display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0088] In this configuration, the user can input commands through the graphical user interface (GUI) displayed on the monitor 260, and the GUI receives the user input commands. Alternatively, the user can input commands by entering specific sounds or gestures, and the GUI receives the user input commands by recognizing the sounds or gestures through a detector.

[0089] In the interactive scenario of the voice-controlled display device shown in this application, the display 260 is configured to display images and a user interface.

[0090] The "user interface" is the medium through which applications or operating systems interact and exchange information with users. It enables the conversion between the internal form of information and the form that users can accept.

[0091] As an example, the user interface can be an icon, window, control, or other interface element displayed on the monitor of display device 200.

[0092] The controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0093] See Figure 3 Taking the deployment of the Android software platform in the display device 200 as an example, the display device 200 divides the Android operating system into four layers, from top to bottom: Applications (referred to as "application layer"), Application Framework layer (referred to as "framework layer"), Android runtime and system library layer (referred to as "system runtime library layer"), and kernel layer.

[0094] In some embodiments, at least one application runs in the application layer. These applications may be Windows, system settings, or clock programs that come with the Android operating system; or they may be applications developed by third-party developers.

[0095] It should be understood that the applications included in this application layer are not limited to the examples above, and may include other applications as well.

[0096] The framework layer provides the application programming interface (API) and programming framework for the application. The application framework layer includes some predefined functions.

[0097] In other words, the framework layer acts as a processing center, determining the actions taken by applications in the application layer. Applications can access Android system resources and obtain system services during execution through API interfaces.

[0098] In some embodiments, the framework layer includes managers, content providers, and a view system.

[0099] The manager includes at least one of the following modules: Activity Manager, which interacts with all activities running on the system; Location Manager, which provides access to system location services for system services or applications; Package Manager, which retrieves various information related to application packages currently installed on the device; Notification Manager, which controls the display and clearing of notification messages; and Window Manager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0100] In some embodiments, the Activity Manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and back of an application; the Window Manager is used to manage all window programs, such as obtaining the screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling changes to the display window (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0101] In some embodiments, the system runtime library layer provides support for the upper layer (i.e., the framework layer). When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0102] In some embodiments, the kernel layer is a layer between hardware and software. The kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0103] After introducing the control interaction scenarios, hardware structure, and deployed software platform of the display device, the following section will provide a detailed explanation of the specific implementation process of the voice control method of the display device based on the aforementioned display device 200.

[0104] With the development of intelligent and information technology, the wider the scenarios and coverage of voice control, the larger its market share will be. Voice functionality is also integrated into various display devices, allowing users to use voice commands to view various multimedia resources and jump to / open specific applications.

[0105] Voice control can be implemented between applications on a display device using direct access words, enabling direct jumps / switches between applications installed on the display device. That is, when the voice control text corresponding to the user's input voice control command is a direct access word for a specific application, the display device responds to the voice control command by jumping from the current application to the target application indicated by the corresponding voice control text.

[0106] As an example, when a user needs to exit the currently browsing video application to the home page, they can instruct the display device to switch from the currently running video application to the home page by entering the voice control command "return to home page," making it easier for the user to select other applications / multimedia resources of interest on the home page.

[0107] It should be understood that the homepage is the user interface that displays and provides direct access to various applications. Users can browse recommended multimedia resources and applications on the homepage to select and play multimedia resources of interest, or open applications of interest to play / browse related content within the application.

[0108] However, for a single application, due to the various scenarios within the application, voice control cannot be achieved across different scenarios using the aforementioned direct access words. Control can only be achieved through interface descriptive words in different display interfaces. That is, when the voice control text corresponding to the user's input voice control command is the interface descriptive word of a control in the current display interface, the display device responds to the voice control command by opening / selecting the target control indicated by the voice control text corresponding to the voice control command in the current display interface.

[0109] Since the application scenarios are relatively fixed, a list of voice interface control words for each display interface within the application can be built during the initial loading process after each application update. This allows the system to quickly identify the target control that the user needs to request an operation on the current display interface after receiving the user's voice control command, based on the list of voice interface control words.

[0110] As an example, after receiving a user's voice control command, the display device matches the voice control text corresponding to the voice control command with the interface description words of each control in the voice interface control word list one by one. If a matching interface description word is found, the target control corresponding to that interface description word is opened / selected in the current display interface.

[0111] In addition, some display interfaces, besides the controls that carry interface descriptors as mentioned above, also include some controls that do not have interface descriptors, such as scroll controls.

[0112] It should be noted that the display interface containing scrolling controls in the display device may include at least one of the following: homepage, webpage, and user interface within an application. This application embodiment does not limit this.

[0113] The scrolling control is used to implement swiping and page-turning operations. Because the scrolling control lacks descriptive text and its scrolling direction is not unique, it is difficult to implement voice control through a list of voice control terms and matching these descriptive texts.

[0114] For ease of understanding, see [link to relevant documentation]. Figure 4 Taking the device homepage as an example, the display interface includes multiple controls such as search, time, VIP, footprint, news, history, featured, and multimedia resources (hereinafter referred to as media assets). Users can select a target control in the control displayed on the display interface through the control device 100, the smart device 300, or voice control commands to open the content display interface of the target control.

[0115] As an example, if the user needs to perform an operation Figure 4 In the displayed interface, control A allows the user to input the voice control command "Activate VIP" into the display device. Upon receiving the voice control command, the display device iterates through the interface descriptions of each control in the voice interface control word list of the display interface according to the voice control text corresponding to the voice control command. After matching the interface description of "Activate VIP" to control A, it operates control A to open the VIP activation content display interface, allowing the user to further perform operations related to activating VIP rights in the content display interface.

[0116] However, scroll controls do not have corresponding interface descriptors. The number of scroll controls contained in a display interface may be different, and the scrolling direction of each scroll control is not unique. Therefore, it is impossible to determine the target scroll control that the user needs to operate from the display interface based on the user's voice control command by matching the interface descriptors mentioned above.

[0117] Continue to see Figure 4 For the scrolling controls B and C in the display interface that do not carry interface descriptions, control B can slide up or down, and control C can slide left or right. Furthermore, control B and control C are only displayed as a slider in the display interface and do not have corresponding interface descriptions. Therefore, it is difficult to determine whether the target scrolling control to be operated is control B or control C based on the user's voice control command in the display interface.

[0118] Based on this, this application provides a voice control method to identify scrolling controls in the current display interface and construct a list of voice scrolling control words for the scrolling controls in the current display interface. This allows the display device 200, upon receiving a user's voice control command, to determine the target scrolling control to be operated on in the current display interface based on the list of voice scrolling control words and the corresponding voice control text, and to control the target scrolling control to perform a scrolling operation according to the scrolling direction indicated by the voice control text. Thus, voice control of scrolling controls in the display device is realized.

[0119] See Figure 5 When the display device 200 executes the voice control method provided in this application, the controller 250 in the display device 200 is configured to perform the following steps:

[0120] Step 510: Determine at least one scroll control contained in the currently displayed interface.

[0121] It should be understood that the current display interface may not have a scroll control, may contain one scroll control, or may contain multiple scroll controls. If the current display interface does not have a scroll control, the user can directly use voice commands to control the corresponding controls to perform operations such as opening / selecting, according to the interface descriptions of the controls on the current display interface.

[0122] If a scrolling control exists in the current display interface, the technical solution provided in this application is executed to determine at least one scrolling control contained in the current display interface, and to construct a list of voice scrolling control words for the scrolling control in the current display interface, so as to realize voice control of the scrolling control in the current display interface to perform scrolling operations.

[0123] As explained above, the current display interface in the embodiments of this application includes at least one of a homepage, a webpage, and an in-application user interface.

[0124] In some embodiments, if the current display interface is the home page, when performing step 510 above, the controller is further configured to: after the display device is powered on, obtain the scrolling controls contained in the home page, and implement voice control of the scrolling controls in the home page to perform scrolling operations through steps 520-530 below.

[0125] In some embodiments, if the current display interface is a webpage, when performing step 510 above, the controller is further configured to: when a jump operation based on a webpage link is detected, use web crawler technology to obtain webpage-related information of the jump webpage, and obtain the webpage content of the webpage by parsing the webpage-related information, thereby determining whether there is a scrolling control in the webpage based on the webpage content.

[0126] As an example, determine the total height of all content items on the webpage based on its content. If the total height of the content items is greater than the window height of the display interface, then a scrolling control exists on the webpage, and the scrolling direction of the scrolling control is vertical scrolling; if the total height of the content items is not greater than the window height of the display interface, then no scrolling control exists on the webpage.

[0127] If it is determined that a scrolling control exists on the webpage, the scrolling operation can be performed by voice control through the following steps 520-530.

[0128] In some embodiments, if the current display interface is a user interface within the application, then when performing step 510 above, the controller is further configured to: after the target application starts, listen for changes in the display interface of the target application; if the display interface changes, obtain control information of multiple controls contained in the changed current display interface; and determine at least one scrolling control contained in the current display interface based on the control information.

[0129] It should be noted that the target application can be a system application that comes with the display device 200, an application that the user installs on the display device 200, or a third-party application that is launched via a network link. This application embodiment does not limit the source and type of the target application.

[0130] In other words, the target application in this application is any application that is triggered and launched by the user on the display device 200.

[0131] In one possible implementation, after the target application on the display device is launched, it registers an Accessibility Service in the framework layer of the Android system. The Accessibility Service can receive some events triggered by the Android system, such as notification status, view-related events, fingerprints, and button clicks (touch).

[0132] Therefore, when the target application starts running, the accessibility service monitors changes to the target application's display interface in real time. When the display interface changes, the accessibility service's callback interface receives a notification of the interface change. In this way, through the interface provided by the accessibility service, the root view node control of the current display interface after the change, as well as the child view node controls corresponding to each root view node control, can be obtained, ultimately yielding all the controls contained in the current display interface.

[0133] Each control in the display device stores control information such as its type and function description. This control information is stored in a structured tree format. Therefore, based on this characteristic, the control information of all controls in the current display interface can be obtained through the structured tree of the current display interface.

[0134] Furthermore, a list of control nodes corresponding to the current display interface can be generated based on the control information of all controls contained in the current display interface.

[0135] It should be noted that the list of control nodes includes controls that have interface descriptions in the currently displayed interface and scroll controls that do not have interface descriptions.

[0136] Therefore, the list of control nodes can be traversed, and the type of each control in the list can be determined to identify the scroll node list corresponding to the currently displayed interface.

[0137] In one possible implementation, the corresponding interface (such as the isScroll interface) is called through the accessibility service to determine whether the control corresponding to each node in the current display interface's structure tree supports scrolling. If scrolling is supported, the control is determined to be a scroll control and added to the scroll control node list.

[0138] In some embodiments, if the scrolling controls included in the display interface are relatively fixed (for example, the display position, type, and functional information of the scrolling controls do not change within a preset time period), then after the scrolling controls corresponding to the display interface are determined for the first time, the correspondence between the display interface and the scrolling control node list can be stored. When the display interface is opened again subsequently, the scrolling controls included in the display interface can be quickly determined based on the scrolling control node list corresponding to the display interface.

[0139] Step 520: Construct a list of voice scrolling control words for the scrolling controls in the currently displayed interface.

[0140] The voice scrolling control word list is used to represent the correspondence between the scrolling direction of the scrolling control and the semantic control words.

[0141] In some embodiments, when performing step 520 above, the controller is further configured to: obtain the display positions of multiple sub-controls controlled by the scroll control in the current display interface; determine the scrolling direction of the scroll control according to the display positions of each sub-control; and construct a list of voice scrolling control words for the scroll control in the current display interface according to the scrolling direction.

[0142] The display position of the child control can be the coordinates of the display area of ​​the child control in the current display interface.

[0143] In one possible implementation, a two-dimensional Cartesian coordinate system is established with the top-left corner / center of the current display interface as the coordinate center. Based on this two-dimensional Cartesian coordinate system, the display positions of each control (scroll control, child controls of scroll control, and other controls carrying interface descriptive words) in the current display interface are determined.

[0144] As an example, the accessibility service calls the corresponding interface (e.g., the getBounds interface) to obtain the display position of each control in the current display interface. The display position can include the boundary coordinates of the control, such as the top boundary coordinates, bottom boundary coordinates, left boundary coordinates, and right boundary coordinates.

[0145] Furthermore, based on the display area coordinates of each sub-control in the current display interface, the display height and display width of each sub-control are determined; then, based on the display height and display width, the scrolling direction of the scroll control to which each sub-control belongs is determined.

[0146] In some embodiments, when determining the scrolling direction of the scroll control, the controller is further configured to:

[0147] (1) If all child controls have the same display height in the current display interface, then the scrolling direction of the scrolling control is determined to be horizontal scrolling.

[0148] The display height is the difference between the coordinates of the control's upper and lower boundaries.

[0149] In other words, when all child controls have the same display height, it means that the multiple child controls are horizontally distributed in the current display interface. Then, the scrolling direction of the scrolling control to which the multiple child controls belong is controlled to be horizontal scrolling, so that the corresponding child controls can be browsed / selected / opened by swiping left and right / turning pages.

[0150] In one example, see Figure 6 Taking the display interface as an example of the user interface within a fitness application, the fitness recommendations in the display interface are scrolling controls, which can control the scrolling of multiple sub-controls corresponding to different fitness items.

[0151] See Figure 6 The fitness recommendations section displays icons for sub-controls such as finding courses, finding plans, fat loss, body shaping training, yoga, AI games, and screen mirroring fitness, as well as icons for other sub-controls provided by the fitness app but not currently displayed on the interface, such as icons for dance and aerobics sub-controls.

[0152] For ease of explanation, Figure 6The scroll control corresponding to the fitness recommendation item is represented by a triangle icon; of course, the scroll control can also be displayed in other forms, and this application embodiment does not limit this.

[0153] In the display interface of this fitness application, since the display height of the icons of multiple fitness items included in the fitness recommendation item is all h, that is, the display height of the multiple sub-controls contained in the scroll control corresponding to the fitness recommendation item is all the same, the scrolling direction of the scroll control corresponding to the fitness recommendation item is horizontal scrolling.

[0154] In this way, by controlling the scrolling control corresponding to the fitness recommendation to perform a horizontal scrolling operation, you can browse multiple fitness programs recommended in the fitness application in turn, and then select the fitness program of interest to practice.

[0155] (2) If all child controls have the same display width in the current display interface, then the scrolling direction of the scrolling control is determined to be vertical scrolling.

[0156] The display width is the difference between the coordinates of the control's left and right boundaries.

[0157] In other words, when the display width of each child control is the same, it means that the multiple child controls are vertically distributed in the current display interface. Then, the scrolling direction of the scroll control to which the multiple child controls belong is controlled to scroll vertically, so that the corresponding child controls can be browsed / selected / opened by sliding up and down / turning pages.

[0158] In one example, see Figure 7 Taking the display interface as an example of a user interface within a music application, the playlist in this display interface (i.e. Figure 7 The XXX chart in the text is a scrolling control that allows you to control the scrolling of multiple sub-controls corresponding to different songs.

[0159] See Figure 7 The playlist displays tabs for songs 01-09, as well as tabs for other sub-controls contained in the playlist but not currently displayed on the screen, such as the tabs for songs 10-121.

[0160] For ease of explanation, Figure 7 The scrolling control corresponding to the playlist is indicated only by the arrow shown on the right; of course, the scrolling control can also be displayed in other forms, and this application embodiment does not limit this.

[0161] In the display interface of this music application, since the display width of the tabs of multiple songs contained in the playlist is d, that is, the display width of the multiple sub-controls contained in the scroll control corresponding to the playlist is the same, the scrolling direction of the scroll control corresponding to the playlist is vertical scrolling.

[0162] Thus, by controlling the scrolling control corresponding to the playlist to perform a scrolling operation in the vertical direction, you can browse all the songs included in the playlist in turn and select the song you are interested in to play.

[0163] (3) If the display height and / or display width of each sub-control are different in the current display interface, then the scrolling direction of the scrolling control is determined to be waterfall scrolling.

[0164] In other words, if the display height of each sub-control is different, or the display width of each sub-control is different, or the display height and display width of each sub-control are different, it means that the multiple sub-controls are distributed in a waterfall layout in the current display interface. In this case, the scrolling direction of the scrolling control to which these multiple sub-controls belong is controlled to be waterfall scrolling, so that multiple sub-controls can be loaded by sliding up and down / turning pages, so that users can browse / select / open the sub-controls they are interested in.

[0165] In one example, see Figure 8 Taking the display interface as an example of the user interface of an image viewing application, the display interface includes a scrolling control that can control the images corresponding to multiple sub-controls. These sub-controls display the corresponding images in a waterfall layout on the display interface.

[0166] See Figure 8 The display interface includes the images corresponding to images 1 through 9, as well as the images corresponding to other sub-controls not shown in the display interface.

[0167] For ease of explanation, Figure 8 The scroll control on the display page is represented only by the scroll bar on the right; of course, the scroll control can also be displayed in other forms, and this application embodiment does not limit this.

[0168] In the display interface of this image viewing application, since images 1-9 are located in different display areas in the current display interface, the boundary coordinates of the child controls corresponding to images 1-9 are different. Therefore, the display height and / or display width of the child controls corresponding to images 1-9 are different, and the scrolling direction of the scrolling control is waterfall scrolling.

[0169] In this way, by controlling the scrolling control in the display page to perform a scrolling operation in the vertical direction, multiple images can be loaded at once after each scrolling operation, allowing users to select images of interest to perform operations such as previewing, downloading, and processing.

[0170] In some embodiments, when constructing the list of voice scrolling control words for the scrolling control in the current display interface, the controller is further configured to: set the semantic control words corresponding to the scrolling control according to the scrolling direction of the scrolling control; and then generate the list of voice scrolling control words for the scrolling control in the current display interface according to the scrolling direction and semantic control words of the scrolling control in the current display interface.

[0171] Among them, semantic control words include swipe semantic words and page-turning semantic words.

[0172] As an example, for a scroll control that scrolls horizontally, the sliding semantic terms set include "swipe left" and "swipe right"; for a scroll control that scrolls vertically, the sliding semantic terms set include "swipe up" and "swipe down"; and for a scroll control that scrolls in a waterfall style, the sliding semantic terms set include "swipe up", "swipe down", "swipe left", and "swipe right".

[0173] As an example, page-turning semantic terms include "previous page" and "next page".

[0174] In some embodiments, assuming that the scrolling controls in the current display interface include M controls, N controls, and P controls, and that the scrolling direction of the M controls is horizontal, the scrolling direction of the N controls is vertical, and the scrolling direction of the P controls is waterfall scrolling, then Table 1 below provides an exemplary list of voice scrolling control words for this current display interface.

[0175] Table 1

[0176]

[0177] Step 530: In response to the user's voice control command, based on the voice scrolling control word list and the voice control text corresponding to the voice control command, control the target scrolling control to perform a scrolling operation in the scrolling direction indicated by the voice control text.

[0178] The target scroll control is the scroll control that corresponds to the voice control text among the scroll controls included in the currently displayed interface.

[0179] In some embodiments, see Figure 9 When step 530 above is implemented, the controller is further configured to perform the following sub-steps:

[0180] Step 531: Obtain the voice control text corresponding to the voice control command.

[0181] Among them, the display device can establish a communication connection between the controller and the cloud server through a communicator, so that the cloud server can parse the voice control commands, thereby reducing the amount of data processing and algorithm storage resources consumed by the display device.

[0182] In some embodiments, when implementing step 531 above, the controller is further configured to: send a voice control command to a cloud server to request the cloud server to parse and process the voice control command; and receive the voice control text corresponding to the voice control command sent by the cloud server.

[0183] In other words, in this embodiment of the application, the voice parsing algorithm is stored in a cloud server. After the cloud server receives the voice control command sent by the display device, it parses the voice control command and obtains the voice control text corresponding to the voice control command.

[0184] In some embodiments, if the cloud server determines that the voice control command is invalid (e.g., no relevant control word character information is parsed) when parsing the voice control command, it sends an abnormal result feedback message to the display device to instruct the display device not to perform any control operation for the time being and to continue to detect the voice control command input by the user.

[0185] Step 533: Match the voice control text with the semantic control words in the voice scrolling control word list to determine the target scrolling control requested by the voice control text in the current display interface, as well as the target scrolling direction of the target scrolling control.

[0186] If there is only one matching scroll control for the voice control text in the current display interface, then the scroll control is directly identified as the target scroll control, and step 535 is executed.

[0187] In some embodiments, if the current display interface contains multiple candidate scroll controls corresponding to voice control text, when performing step 533 above, the controller is further configured to: obtain the control priority of each candidate scroll control; and determine the target scroll control from the multiple candidate scroll controls according to the control priority.

[0188] In one possible implementation, the control priority of each candidate scroll control is determined based on the control relationship between each candidate scroll control and the display position of each candidate scroll control in the current display interface.

[0189] In the current display interface's corresponding tree structure, the scroll control located at the parent node has a higher priority than the scroll control located at the child node.

[0190] Among them, the scroll control that is displayed in the upper left corner of the current display interface has a higher priority than the scroll control that is displayed in the lower right corner of the current display interface.

[0191] As an example, see Figure 10 The display interface shown includes multiple scrolling controls: W control, X control, Y control, and Z control. The W control can control the X, Y, and Z controls, therefore the W control has higher priority than the X, Y, and Z controls.

[0192] The control priorities of the X, Y, and Z controls can be further determined based on their display positions on the screen. The X control is located at the top of the screen and has the highest priority; the Z control is located at the bottom of the screen and has the lowest priority. Therefore, the control priorities of the X, Y, and Z controls are: X > Y > Z.

[0193] therefore, Figure 10 The control priority of the four scrolling controls is: control W > control X > control Y > control Z.

[0194] Thus, once the control priority of the scrolling controls in the current interface is determined, a target scrolling control can be selected from the voice control commands and the list of voice scrolling control words to perform the scrolling operation.

[0195] It should be understood that after receiving a user's voice control command, the controller can determine the control requested by the user in the current display interface in the order of "first the voice interface control word list, then the voice scroll control word list"; the controller can also determine the control requested by the user in the current display interface in the order of "first the voice scroll control word list, then the voice interface control word list". This application embodiment does not limit this.

[0196] In some embodiments, if it is determined that the current display interface contains a scrolling control, after the display device receives the user's voice control command, it can first determine, based on the voice scrolling control word list, whether the voice control text corresponding to the voice control command exists in the current display interface as the target scrolling control requested.

[0197] Furthermore, if a target scrolling control matching the voice control text cannot be determined based on the voice scrolling control word list, then the system continues to determine whether a target control requesting control exists in the voice interface control word list of the currently displayed interface, based on the voice control text. The interface description word of the target control in the voice interface control word list is the same as the voice control text.

[0198] Step 535: Based on the voice control text, control the target scroll control to perform a scrolling operation in the target scrolling direction.

[0199] In some embodiments, when implementing step 535 above, the controller is further configured to: obtain the target scrolling distance of the target scrolling control based on the voice control text; and control the target scrolling control to perform a scrolling operation in the target scrolling direction according to the target scrolling distance.

[0200] The voice control text can include swipe-based voice text and page-turning voice text.

[0201] In one possible implementation, if the voice control text is swiping voice text, the scrolling distance can be a preset distance value. For example, the above distance value can be set according to the display height, display width, display position interval, etc. of multiple sub-controls controlled by the scrolling control. This application embodiment does not limit this.

[0202] As an example, if the voice control text is gliding voice text, see [link to relevant documentation]. Figure 6 The scroll distance of the scroll control can be the display position interval between the "Find Courses" icon and the "Find Schedules" icon; see also Figure 7 The scroll distance of the scroll control can be the display width of the song 01 tab, i.e., d; see [link / reference]. Figure 8 The scroll distance of the scroll control can be the display width or display height of any image in the display interface.

[0203] In some embodiments, if the voice control text is page-turning voice text, the controller is further configured to calculate the scroll distance as follows: if the scrolling direction is horizontal, the scroll distance is calculated based on the display positions of the rightmost and leftmost sub-controls controlled by the target scroll control; if the scrolling direction is vertical or waterfall scrolling, the scroll distance is calculated based on the display positions of the topmost and bottommost sub-controls controlled by the target scroll control.

[0204] As an example, if the voice control text is page-turning voice text, see [link to relevant documentation]. Figure 6 The scroll distance of the scroll control can be the total display width between the "Find Courses" icon and the "Cast Training" icon; see also Figure 7 The scroll distance of the scroll control can be the total display height between the tabs for songs 01 to 09, i.e., 9*d; see [link / reference]. Figure 8 The scroll distance of the scroll control can be the total display height or total display width of multiple images in the current display interface. For example, the scroll distance can be the total display height of images 2 and 6 in the display interface.

[0205] Based on the above, the controller in the display device provided in this application is connected to both the display and the sound acquisition device. During the display of images and the user interface, the controller identifies at least one scrolling control in the current display interface and constructs a list of voice scrolling control words for that scrolling control. This list represents the correspondence between the scrolling direction of the scrolling control and the semantic control words. Further, after detecting and acquiring a user's voice control command, the sound acquisition device sends the command to the controller. The controller responds to the command by controlling the target scrolling control corresponding to the voice control command in the current display interface to perform a scrolling operation based on the pre-constructed list of voice scrolling control words and the corresponding voice control text. Thus, by recognizing scrolling controls and constructing a list of voice scrolling control words for them, the display device can control the target scrolling control in the current display interface to perform a scrolling operation according to the user's voice control command when it receives the command. This overcomes the limitation that scrolling controls cannot be directly controlled by voice, achieving full voice control of all controls in the display device, improving the flexibility and convenience of the display device's control method, and enhancing the user experience.

[0206] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0207] In addition, this application also provides a voice control method, which can be applied to the above-mentioned display device 200 or other electronic devices, the method comprising:

[0208] Identify at least one scroll control contained in the currently displayed interface;

[0209] Construct a list of voice scrolling control words for the scrolling controls in the current display interface; the list of voice scrolling control words is used to represent the correspondence between the scrolling direction of the scrolling controls and the semantic control words;

[0210] In response to the user's voice control command, based on the list of voice scrolling control words and the corresponding voice control text, the target scrolling control is controlled to perform a scrolling operation in the scrolling direction indicated by the voice control text; the target scrolling control is one of the scrolling controls in the currently displayed interface that corresponds to the voice control command.

[0211] In some embodiments, see Figure 11 The voice control logic for the scrolling control is as follows: After the target application starts, an Accessibility Service is registered to monitor page changes. This service monitors page changes. When a control on the display page changes (i.e., the display page changes), the root view node control and its child view node controls are retrieved based on the control structure tree. Finally, a list of control nodes corresponding to the current display interface is generated based on the root view node control and its child view node controls.

[0212] Furthermore, the list of control nodes in the current display interface is traversed to determine at least one scroll control contained in the current display interface, and a list of scroll control nodes corresponding to the current display interface is generated.

[0213] Furthermore, for any given scroll control, obtain the display positions of the multiple sub-controls controlled by that scroll control, such as the coordinates of the control's top boundary, bottom boundary, left boundary, and right boundary; determine the scrolling direction of the scroll control based on the display positions of each sub-control controlled by the scroll control, and set the corresponding semantic control words for the scroll control; generate a list of voice scrolling control words for the scroll controls in the current display interface based on the scrolling direction and semantic control words of each scroll control.

[0214] If a user's voice control command is received, the display device will, based on the voice control text corresponding to the voice control command, traverse the list of voice scrolling control words to determine the target scrolling control corresponding to the voice control text, and then control the target scrolling control to perform scrolling operations.

[0215] It should be noted that when there are multiple scroll controls in the current display interface, the target scroll control in the current display interface will be controlled to perform the scrolling operation according to the control priority.

[0216] For details on the implementation principle and beneficial effects of this voice control method, please refer to the specific limitations of the various embodiments regarding the controller configuration content in the display device mentioned above, which will not be repeated here.

[0217] In some embodiments, this application also provides a computer-readable storage medium. This computer-readable storage medium may store a computer program that is invoked and executed by a controller in a display device or other electronic device to implement some or all of the steps of the voice control method provided in this application.

[0218] As an example, the computer-readable storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0219] It should be understood that the technical solutions in the embodiments of this application can be implemented using software plus the necessary general-purpose hardware platform. Therefore, the technical solutions in the embodiments of this application, in essence or in part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium.

[0220] In some embodiments, this application also provides a computer program product. This computer program product includes a computer program that is invoked and executed by a controller in a display device or other electronic device, and can implement some or all of the steps of the voice control method provided in this application.

[0221] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be made based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the contents of this disclosure, thereby enabling those skilled in the art to better use the embodiments.

[0222] In other words, any modifications, equivalent substitutions, or improvements made by those skilled in the art without departing from the technical concept of this application should be included within the scope of protection of this application.

Claims

1. A display device, characterized in that, include: The monitor is configured to display the display interface; The voice acquisition device is configured to collect the user's voice control commands; A controller connected to the display and the sound acquisition unit respectively, wherein the controller is configured to: Obtain at least one scrolling control contained in the display interface, and the display positions of multiple sub-controls controlled by the scrolling control in the display interface; The scrolling direction of the scrolling control is obtained based on the display positions of the multiple sub-controls; Based on the scrolling direction, a list of voice scrolling control words for the scrolling control in the display interface is constructed. The list of voice scrolling control words is used to represent the correspondence between the scrolling direction of the scrolling control and the semantic control words. In response to a user's voice control command, based on the list of voice scrolling control words and the voice control text corresponding to the voice control command, the target scrolling control is controlled to perform a scrolling operation in the scrolling direction indicated by the voice control text; the target scrolling control is a scrolling control among the scrolling controls included in the display interface that corresponds to the voice control text.

2. The display device according to claim 1, characterized in that, The controller is further configured to acquire at least one scrolling control included in the display interface. After the target application is launched, monitor the changes in the target application's display interface; If the display interface changes, obtain the control information of the multiple controls contained in the changed display interface; Based on the control information, obtain at least one scrolling control contained in the display interface.

3. The display device according to claim 1, characterized in that, The controller executes the process of obtaining the scrolling direction of the scrolling control based on the display positions of the multiple sub-controls, and is further configured to: If multiple sub-controls have the same display height in the display interface, then the scrolling direction of the scrolling control is determined to be horizontal scrolling. If multiple sub-controls have the same display width in the display interface, then the scrolling direction of the scrolling control is determined to be vertical scrolling. If multiple sub-controls have different display heights and / or display widths in the display interface, then the scrolling direction of the scrolling control is determined to be waterfall scrolling.

4. The display device according to any one of claims 1 to 3, characterized in that, The controller executes voice control text based on the voice scrolling control word list and the voice control command, and controls the target scrolling control to perform a scrolling operation according to the scrolling direction indicated by the voice control text, and is further configured as follows: Obtain the voice control text corresponding to the voice control command; The voice control text is matched with the semantic control words in the voice scrolling control word list to determine the target scrolling control requested by the voice control text in the display interface, and the target scrolling direction of the target scrolling control. Based on the voice control text, the target scroll control is controlled to perform a scrolling operation in the target scrolling direction.

5. The display device according to claim 4, characterized in that, The controller, which executes the voice control text to control the target scroll control to perform a scrolling operation in the target scrolling direction, is further configured to: Based on the voice control text, obtain the target scrolling distance of the target scrolling control; The target scroll control is controlled to perform a scrolling operation in the target scrolling direction according to the target scrolling distance.

6. The display device according to claim 5, characterized in that, The voice control text includes page-turning voice text, and the controller executes the process of obtaining the target scrolling distance of the target scrolling control based on the voice control text, which is further configured as follows: If the target scrolling direction is horizontal, the target scrolling distance is calculated based on the display positions of the rightmost and leftmost sub-controls controlled by the target scrolling control. If the target scrolling direction is vertical scrolling or waterfall scrolling, the target scrolling distance is calculated based on the display positions of the topmost and bottommost sub-controls controlled by the target scrolling control.

7. The display device according to claim 4, characterized in that, If the display interface contains multiple candidate scrolling controls corresponding to the voice control text, the controller is further configured as follows: Obtain the control priority of the candidate scroll control; The target scroll control is selected from the plurality of candidate scroll controls according to the control priority.

8. The display device according to claim 4, characterized in that, The controller executes the process of obtaining the voice control text corresponding to the voice control command, and is further configured as follows: The voice control command is sent to the cloud server to request the cloud server to parse and process the voice control command. Receive the voice control text corresponding to the voice control command sent by the cloud server.

9. A voice control method, characterized in that, include: Obtain at least one scrolling control contained in the display interface, and the display positions of multiple sub-controls controlled by the scrolling control in the display interface; The scrolling direction of the scrolling control is obtained based on the display positions of the multiple sub-controls; Based on the scrolling direction, a list of voice scrolling control words for the scrolling control in the display interface is constructed. The list of voice scrolling control words is used to represent the correspondence between the scrolling direction of the scrolling control and the semantic control words. In response to a user's voice control command, based on the list of voice scrolling control words and the voice control text corresponding to the voice control command, the target scrolling control is controlled to perform a scrolling operation in the scrolling direction indicated by the voice control text; the target scrolling control is a scrolling control among the scrolling controls included in the display interface that corresponds to the voice control text.

Images