Interface control method, device, and system

Abstract

The present application provides an interface control method, which includes the steps of: obtaining a user's voice command and the user's sound source position; obtaining the user's gaze information; determining a target window on an interface according to the sound source position and the gaze information; and controlling the target window according to the voice command. According to the interface control method in the present application, collaborative decision-making is performed with reference to multi-modal information such as sound source information, gaze tracking information, audio semantic information, and their priorities, so that page content in multiple windows on the interface is quickly and accurately controlled, and the user experience is improved.

Description

[Technical field]

[0001] TECHNICAL FIELD The embodiments of the present application relate to the field of human-machine interaction, and more particularly to an interface control method and apparatus and system. [Background technology]

[0002] With the development of science and technology, smart head units have richer functions, such as navigation, mobile phone interconnection, voice control, built-in applications (APP), and in-vehicle network. This brings users a more convenient and intelligent driving experience. The head unit interface is the main medium for interaction between the user and the intelligent vehicle. The user may use touch operation or voice command to control the head unit interface to implement interaction with the intelligent vehicle. When the user uses touch operation to control the head unit interface, the user needs to lean the body to perform the operation. Working with complex procedures takes time and increases the risk of accidents. When the user uses voice command to control the head unit interface, driving safety can be ensured. Therefore, it is particularly important to improve the user experience when the user uses voice command to control the head unit interface. Summary of the Invention [Means for solving the problem]

[0003] The embodiments of the present application provide an interface control method and apparatus, which performs collaborative decision-making by referring to multi-modal information, such as audio source information, eye-tracking information, audio-semantic information, and their priorities, so as to quickly and accurately control page contents in multiple windows on an interface, thereby improving user experience.

[0004] According to a first aspect, an interface control method is provided, the method includes: acquiring a user's voice command and a user's sound source position, acquiring user's gaze information, determining a target window on an interface based on the sound source position and the gaze information, and controlling the target window based on the voice command.

[0005] According to the interface control method in the present application, the target window to be controlled by the user is determined using multi-modal information such as the user's voice semantics, the user's sound source position, and the user's gaze information, etc. The target window is then controlled based on the voice command, so that the user can precisely control any window on multiple head unit interfaces to improve the user experience.

[0006] In some possible implementations, the interest window is located closest to the sound source position and in the line of sight direction indicated by the line of sight information.

[0007] According to the interface control method of the present application, both the user's sound source position and the user's gaze information are taken into account, so that the determined target window is the window that should actually be controlled by the user, thereby improving accuracy.

[0008] In some possible implementations, the window closest to the sound source position is the first window, and the window located in the gaze direction indicated by the gaze information is the second window. Determining a target window on the interface based on the sound source position and the gaze information includes determining a target window based on a priority of the sound source position and a priority of the gaze information. If the priority of the sound source position is higher than the priority of the gaze information, the first window becomes the target window, and if the priority of the gaze information is higher than the priority of the sound source position, the second window becomes the target window.

[0009] In some possible implementations, the priority of the sound source position and the priority of the gaze information are predefined.

[0010] In some possible implementations, the sound source position has first priority information and the line of sight information has second priority information, the first priority information being used to determine the priority of the sound source position and the second priority information being used to determine the priority of the line of sight information.

[0011] In some possible implementations, the method includes obtaining an execution result of the voice command, and adjusting the first priority information and the second priority information based on the execution result.

[0012] In some possible implementations, there is a first correlation between the sound source location and the service indicated by the voice command, and there is a second correlation between the sound source location and the service indicated by the voice command.

[0013] In some possible implementations, the method further includes adjusting the first correlation and the second correlation based on an execution result of the voice command.

[0014] According to the interface control method in the present application, it may be further considered that when the target window determined separately based on the user's semantics, sound source position, and gaze information does not match, the target window can be determined based on the priority of semantics, sound source position, and gaze information of the user. This can avoid the case where the user's voice command cannot be executed due to the target window determined separately based on the user's semantics, sound source position, and gaze information not matching. Furthermore, the priority of the user's semantics, sound source position, and gaze information can be adjusted based on the execution result to better adapt to the user's habits.

[0015] In some possible implementations, controlling the target window based on the voice command includes displaying, in the target window, an icon corresponding to a service indicated by the voice command, the icon including one or more indexes.

[0016] In some possible implementations, the method further includes displaying an icon corresponding to the first service in the target window, the icon including one or more indexes. The voice command indicates the target index, and the target index belongs to the one or more indexes. Controlling the target window based on the voice command includes executing a target control of the first service and indicated by the target index.

[0017] The interface control method in the present application further includes displaying an icon index in the target window, which is simpler and more eye-catching than specific text information, making it easier for users to recognize and select, avoiding users from being too distracted, and improving driving safety.

[0018] According to a second aspect, there is provided an interface control method, comprising the steps of: receiving a first voice command of a user, the first voice command indicating a first service, displaying an icon of the first service in a target window, the icon including a plurality of indexes, receiving a second voice command of the user, the second voice command indicating a target index, the target index belonging to the plurality of indexes, and executing a task of the first service and corresponding to the target index.

[0019] In some possible implementations, the target window is the window closest to the user or the window in the user's line of sight.

[0020] According to a third aspect, an interface control device is provided. The interface control device includes an acquisition unit configured to acquire a user's voice command and a user's sound source position, the acquisition unit being further configured to acquire the user's gaze information, and a processing unit configured to determine a target window on the interface based on the sound source position and the gaze information. The processing unit is further configured to control the target window based on the voice command.

[0021] In some possible implementations, the interest window is located closest to the sound source position and in the line of sight direction indicated by the line of sight information.

[0022] In some possible implementations, the window closest to the sound source position is the first window, and the window located in the line of sight direction indicated by the line of sight information is the second window. The processing unit is particularly configured to determine the target window based on the priority of the sound source position and the priority of the line of sight information. If the priority of the sound source position is higher than the priority of the line of sight information, the first window is the target window, or if the priority of the line of sight information is higher than the priority of the sound source position, the second window is the target window.

[0023] In some possible implementations, the priority of the sound source position and the priority of the gaze information are predefined.

[0024] In some possible implementations, the sound source position has first priority information and the line of sight information has second priority information, the first priority information being used to determine the priority of the sound source position and the second priority information being used to determine the priority of the line of sight information.

[0025] In some possible implementations, the apparatus is further configured to obtain an execution result of the voice instruction, and adjust the first priority information and the second priority information based on the execution result.

[0026] In some possible implementations, there is a first correlation between the sound source location and the service indicated by the voice command, and there is a second correlation between the gaze information and the service indicated by the voice command.

[0027] In some possible implementations, the apparatus is further configured to adjust the first correlation and the second correlation based on an outcome of execution of the voice command.

[0028] In some possible implementations, the execution unit is specifically configured to display an icon in the target window corresponding to the service indicated by the voice command. The icon includes one or more indexes.

[0029] In some possible implementations, the apparatus is further configured to display an icon corresponding to the first service in the object window, the icon including one or more indexes. The voice command indicates the object index, the object index belonging to the one or more indexes. Controlling the object window based on the voice command includes executing the object control of the first service and indicated by the object index.

[0030] According to a fourth aspect, an interface control device is provided. The interface control device includes an acquisition unit configured to acquire a first voice command of a user, the first voice command indicating a first service, and a processing unit configured to display an icon of the first service in a target window, the icon including a plurality of indexes. The acquisition unit is further configured to acquire a second voice command of the user, the second voice command indicating a target index, the target index belonging to the plurality of indexes. The processing unit is of the first service and is further configured to execute a task corresponding to the target index.

[0031] In some possible implementations, the target window is the window closest to the user or the window in the user's line of sight.

[0032] According to a fifth aspect, there is provided an interface control device, the interface control device including a processor and a memory, the memory configured to store program instructions, the processor configured to invoke the program instructions to perform the method according to any one of the embodiments of the first and second aspects.

[0033] According to a sixth aspect, there is provided an interface control device, the interface control device including a processor and a data interface, the processor reading instructions stored in a memory via the data interface to execute the method according to any one of the embodiments of the first and second aspects.

[0034] According to a seventh aspect, there is provided a vehicle comprising an apparatus according to any one of the embodiments of the third and fourth aspects.

[0035] According to the interface control method in the present application, a target window to be controlled by a user is determined using multi-modal information such as a user's voice semantics, a user's sound source position, and a user's gaze information. The target window is then controlled based on a voice command, so that the user can accurately control any window on multiple head unit interfaces to improve the user experience. According to the interface control method in the present application, it may be further considered that if the target window determined separately based on the user's semantics, sound source position, and gaze information does not match, the target window may be determined based on the priority of the semantics, sound source position, and the user's gaze information. This can avoid a case where the user's voice command cannot be executed due to the target window determined separately based on the user's semantics, sound source position, and gaze information not matching. Furthermore, the priority of the user's semantics, sound source position, and gaze information can be adjusted based on the execution result to better adapt to the user's habits. The interface control method in the present application further includes a step of displaying an index of an icon in the target window. Icon indexes are simpler and more eye-catching than concrete text information, making them easier for users to recognize and select, avoiding excessive user distraction, and improving driving safety. [Brief description of the drawings]

[0036] [Figure 1] 1 is a schematic diagram of an interior structure of a vehicle according to an embodiment of the present application; [Diagram 2] FIG. 2 is a schematic diagram of an example of an application architecture of an interface control method according to an embodiment of the present application; [Diagram 3] FIG. 2 is a schematic diagram of an exemplary module in an interface control method according to an embodiment of the present application. [Figure 4] 4 is a schematic flowchart of determining an interest window based on user semantics, sound source location, and gaze information according to an embodiment of the present application; [Diagram 5] 1 is a schematic flowchart of an interface control method according to an embodiment of the present application. [Figure 6] 2 is a schematic diagram of three possible icons according to an embodiment of the present application. [Figure 7] FIG. 2 is a schematic diagram of a driver's seat user's control of a head unit interface according to one embodiment of the present application. [Figure 8] FIG. 2 is a schematic diagram of a passenger user's control of a head unit interface according to one embodiment of the present application. [Figure 9] FIG. 2 is a schematic diagram of a driver's seat user's control of a rear row head unit interface according to one embodiment of the present application. [Figure 10] FIG. 2 is a schematic diagram of an interface having multiple windows according to an embodiment of the present application. [Figure 11] 2 is a schematic flow chart of controlling an interface using voice commands by a user according to an embodiment of the present application; [Figure 12] 4 is a schematic flowchart of another interface control method according to an embodiment of the present application. [Figure 13] FIG. 2 is a schematic block diagram of an interface control device according to an embodiment of the present application. [Figure 14] FIG. 2 is a schematic block diagram of an interface control device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] The following describes the technical solutions of the embodiments in this application with reference to the accompanying drawings.

[0038] Intelligent terminals are more and more widely applied in life. For example, the intelligent terminal may be a smart phone, various intelligent appliances in a smart home, or an intelligent vehicle. These intelligent devices are more and more widely used in people's lives. In order to facilitate the interaction between people and intelligent terminals, a display interface is usually arranged on the intelligent terminal to conveniently display quick information or information required by the user, such as text information, image information, or video information, to the user. Therefore, with the widespread use of intelligent terminals, a scenario appears in which multiple display interfaces exist in the same space. The display interface may be the display interface of different intelligent terminals or the display interface of the same intelligent terminal. For example, a flip phone has at least two display screens, and audio devices and video devices in a smart home may have multiple display screens, or multiple display screens may be provided to a user in the cockpit of an intelligent vehicle. Thus, the use requirements of different passengers are met. The interface is controlled using voice commands to free the user's hands. In this case, the user can control the intelligent terminal without touching the intelligent terminal, and the user experience is improved. Therefore, using voice commands to control an intelligent terminal has become the main human-machine interaction method. In addition, using voice commands to control the head unit interface (i.e., the display screen in the cockpit) can improve driving safety compared with using touch operations, so voice commands have become the main way to control the head unit interface in the driving process.

[0039] In the existing method of using voice commands to control an intelligent terminal, semantic analysis is performed on the voice command to determine the action indicated by the voice command. However, with the increasing scenario of multiple display interfaces in the same space, in this voice control method, the display interface for executing the voice command is usually different from the display interface that the user actually expects to execute the voice command, resulting in a poor user experience.

[0040] For example, more intelligent terminals use a large screen or a long screen, respectively. The large screen or the long screen may correspond to a split screen mode. Specifically, the large screen or the long screen may be used as one display interface or may be divided into multiple display interfaces for use by different users. As an example, a display screen in a vehicle cockpit is used. A long screen may be placed in front of the driver and the passenger in the front seat. The long screen corresponds to a split screen mode. The split screen mode may be understood as a scenario that the head unit system supports. Specifically, multiple application areas (also called windows or display interfaces) are simultaneously displayed on one screen. Here, as an example, two application areas are used, which are used for the driver and the passenger in the front seat to operate their respective applications independently. When an interface is divided into a primary screen (also called a primary application area, a primary window, a primary display interface, etc.) and a secondary screen (also called a secondary application area, a secondary window, a secondary display interface, etc.), it cannot be determined based on existing voice control technology whether a user's voice command operates on the primary screen or the secondary screen. In addition, when there are multiple display interfaces in the same space, existing voice control technology is not adaptable enough. For example, only control on the focus interface can be supported. The focus interface is the interface that is closest to the user and is controlled using voice commands. As a result, the user cannot flexibly control the required interface (e.g., an interface far away from the user) using voice commands, resulting in a poor user experience.

[0041] Therefore, an embodiment of the present application provides an interface control method, in which collaborative decision-making is performed with reference to multi-modal information, so that the content on the interface required by the user is more accurately controlled, improving the user experience. The multi-modal information may include at least two kinds of information, such as sound source position information, user gaze information, semantic information of voice command, and interface display information.

[0042] In the following embodiments of the present application, the interface control method is described using the scenario of a vehicle intelligent cockpit as an example. However, the embodiments of the present application are not limited thereto. The method may also be applied to other scenarios in which split-screen control or multi-interface control is performed, such as control scenarios of smart home appliances, smartphones, etc.

[0043] FIG. 1 is a schematic diagram of an internal structure of a vehicle according to an embodiment of the present application. In a vehicle, an in-vehicle terminal such as a head unit (also called an in-vehicle audio and video entertainment system) may be disposed in the center console of the vehicle, and the screen of the center console may also be called a central display screen or a central display screen. In addition, some luxury cars are gradually applying comprehensive digital displays in the cockpit. One or more display screens are disposed in the cockpit and configured to display contents such as a digital dashboard and an in-vehicle entertainment system. As shown in FIG. 1, multiple display screens are disposed in the cockpit, for example, a digital dashboard display screen 101, a central display screen 102, and a display screen 103 in front of a passenger seat passenger (also called a front seat passenger). The display screen 103 may be a split screen. As shown in FIG. 1, the display screen 103 is divided into two screens, a display screen 104 in front of a left rear seat passenger and a display screen 105 in front of a right rear seat passenger. Further, although only one camera 106 is shown in FIG. 1 near the A-pillar on the driver's side, multiple cameras may be arranged in the cockpit, and the position of the camera is free. For example, some cameras in the cockpit may be arranged above the central display screen of the vehicle, some cameras in the cockpit may be arranged to the left of the central display screen of the vehicle, some cameras in the cockpit may be arranged on the A-pillar or B-pillar, and some cameras in the cockpit may be arranged in front of the cockpit top of the vehicle. It should be understood that the vehicle further includes one or more microphones (not shown in FIG. 1), and the microphones may be arranged above the central display screen in the vehicle or in front of the cockpit top in the vehicle. In the interface control method in the embodiment of the present application, the camera may acquire the user's gaze information, behavior information, etc., and the microphone may acquire the user's voice information, and perform collaborative decision-making based on the acquired information about the user. This allows the user to control multiple display screens in the vehicle.

[0044] FIG. 2 is a schematic diagram of an application architecture of an interface control method according to an embodiment of the present application. A scenario of a head unit application is used as an example. As shown in FIG. 2, the head unit system includes a display device, a voice sensor, a voice player, and a processing module. The processing module includes a voice signal processing module (sometimes also called a voice service module) and a decision module. A voice assistant application may be installed in the head unit, and the head unit uses the voice signal processing module to perform voice interaction with a user. After the head unit acquires the user's voice, the voice signal processing module performs operations such as language understanding and semantic recognition on the user's voice, converts the user's voice into an instruction that can be recognized by the head unit, and executes the instruction. In addition, the voice signal processing module may generate a corresponding response based on a specific execution operation, perform voice synthesis on the response, and then report the response to the user via dialogue management to respond to the user's voice command. In this way, a multi-round voice interaction between the head unit and the user is implemented. The head unit may acquire the user's voice signal using a voice sensor, such as a microphone. In addition, the head unit may output the voice signal using a speech player, such as a speaker. In this process, the user can use voice to control the head unit interface. The decision module is configured to implement collaborative decision-making based on multi-modal information (shortly referred to as multi-modal decision-making). The decision module collaboratively determines the user's intention and an interface corresponding to the intention based on the multi-modal information, and then processes the user's intention to implement an operation on the corresponding interface. The multi-modal information is the same as described above. The details will not be described again here. The decision module may further include an interface decision unit and an intention processing unit. The interface decision unit is configured to determine an interface corresponding to the user's intention based on at least one kind of information of the multi-modal information.The intention processing unit is configured to determine a user's intention based on at least one type of information of the multi-modal information, and process the user's intention, including at least one of tapping a control on the interface (this intention is referred to as a control tapping type intention), a general command action (this intention is referred to as a general command type intention), intention generalization, conflict processing, etc. The general command is at least one of a set of global interface control commands, such as flick up (or down), flip up (or down), return, and back, input by voice when the interface is in a flickable or flippable state. The intention generalization is the ability to perform corpus expansion and knowledge graph to a certain degree. For example, if the tappable action poster on the interface is xx movie, the user can add some prefixes and say, for example, "I want to watch xx movie" or "help me open xx movie". Alternatively, the user can add some suffixes and say, for example, "help me open xx video" or some suffix clichés generalized. The knowledge graph indicates content related to the target content. For example, if the action poster on the current interface is a xx movie, when the user says the director or leading actor of the xx movie, a corresponding action can be performed on the xx movie on the interface. For example, information about the director or leading actor of the xx movie is displayed or played. Conflict processing is processing that is performed when a conflict occurs when a user's voice command is executed on the current interface. For example, for a window that is open on the current interface, when the user says "close" in a voice command, the voice command indicates tapping the close button of the window on the current interface or indicates a global command to close the entire application. In this case, the intention processing unit may process these commands that may cause a conflict. After the user's intention is converted into a specific command, an action on the application is executed based on the command.The head unit system may further include a screen sensing module (sometimes also referred to as an interface sensing module) configured to sense at least one of interface information in the multi-modal information, such as attributes of a control on the interface, such as text displayed on the control, coordinates of the control, and whether the control can be tapped, etc. It can be seen that the screen sensing module is configured to provide interface information in the multi-modal information.

[0045] An application control information acquisition module is further included between the decision module and the application in FIG. 2. The decision module in FIG. 2 may specifically be divided into a decision processing module and a decision execution module. FIG. 3 shows three modules. The application control information acquisition module and the decision execution module may be disposed on the terminal side, and the decision processing module may be disposed on the terminal side or the cloud side. It should be understood that when the application control information acquisition module, the decision execution module, and the decision processing module are disposed on the terminal side, the application control information acquisition module, the decision execution module, and the decision processing module may specifically be a processor (e.g., an intelligent device) on the terminal side. When the decision processing module is disposed on the cloud side, the decision processing module may specifically be a server on the cloud side, etc. The server may be a physical server or a virtual server. The application control information acquisition module is configured to acquire control information of the application, including control layout, control content, control attributes, etc., for example, text information, icons, switches, players, and input / search. The acquisition manner includes an application report, a screen detection module, etc. The application control information acquisition module inputs the collected interface information, gaze tracking information, sound source position information, and semantic information together into the see-and-speak voice processing module. The decision processing module is configured to determine a target window, and perform intent matching with reference to the application control information of the target window to determine a control command. The target control determination includes determining a target window based on the sound source position and gaze information. The intent matching with reference to the application control information of the target window includes processing control descriptions in the control information, converting near-syllabic words, performing digital conversion, and processing special characters. Finally, the determined control command includes commands such as tapping the control, switching list pages, on / off, and flicking. The decision processing module is usually deployed on a cloud.In order to improve the command execution speed in applications such as head units, the decision processing module is deployed on the terminal side, which greatly improves the command execution experience in scenarios where the network is insufficient during the driving process. After receiving the execution packet delivered by the decision module, the decision execution module selects a corresponding execution scheme based on the specific execution packet, including tap (e.g., single tap, double tap, long press, etc.), back (including returning to the previous page, returning to the menu, returning to the home page, etc.), and flick (flick up, flick down, left flick, right flick, etc.).

[0046] 4 is a schematic flowchart of determining a target window (or interface) based on a user's semantics, sound source position, and gaze information according to an embodiment of the present application. As shown in FIG. 4, after a user's voice command is obtained, a target window and a target service are determined based on the user's semantic information, sound source position, and / or gaze information, and an operation related to the target service in the target window is performed.

[0047] First, a user's voice command is obtained, and then semantic recognition is performed based on the user's voice command to obtain the user's semantics. Here, the user's voice command may be recognized using multiple kinds of semantic recognition techniques, for example, using deep fully convolutional neural network (DFCNN), LFR-DFSMN (lower frame rate deep feedforward sequential memory networks), or streaming truncated multi-layer attention (SMLTA).

[0048] Then, whether the user directly expresses the subjective intention using the voice command or whether the user directly indicates the target window using the voice command may be determined based on the user's semantics. For example, the head unit interface is divided into a driver screen and a passenger screen. If the user's voice command is "open music on the driver screen", the user's semantics clearly defines the subjective intention. Specifically, the target window to be controlled is the driver screen. The driver screen is directly controlled based on the user's subjective intention. Optionally, the user's semantic information of the multi-modal information may be set to take precedence over the user's other information. However, in practical applications, the user's other information may alternatively be set to take precedence over the semantic information, or the priority of the semantic information may be set adjustable by the user. In other words, in the application process, the user may adjust the priority of the semantic information based on the actual requirements.

[0049] If the user's voice command does not express a subjective intention, the target window to be controlled cannot be directly determined from the user's semantics. In this case, the target window to be controlled by the user is determined based on other information of the user, for example, based on the user's sound source position and / or the user's gaze information. Alternatively, it may not be determined whether the user's voice command expresses a subjective intention, and the target window to be controlled by the user is directly determined based on the user's sound source position and / or the user's gaze information. For a method for determining a target window to be controlled by the user based on the user's sound source position and / or the user's gaze information, please refer to the description of FIG. 5 below.

[0050] After the target window to be controlled by the user is determined, the user's voice command is executed in the target window. Specifically, the user's control intention is obtained based on the user's voice command, and then the user's control intention is executed in the target window based on the obtained control information of the application, such as tapping, flicking, turning a page, etc. Then, it is determined whether the user's voice command is successfully executed in the target window. If the user's voice command is successfully executed in the target window, dotting is performed to record that the voice command is successfully executed. Optionally, if the user's voice command is not executed in the target window, self-learning optimization is performed. For example, the priority of the sound source position and the priority of the gaze information are adjusted based on the execution result of the voice command. For details, please refer to the following description of step 504 in FIG. 5.

[0051] With reference to FIG. 5, the following describes an interface control method based on a user's sound source position and / or a user's gaze information according to an embodiment of the present application.

[0052] 5 is a schematic flowchart of an interface control method 500 based on a user's sound source position and / or a user's gaze information according to an embodiment of the present application. As shown in FIG. 5, the method includes steps 501 to 504, which are described separately below. For ease of explanation, an example is used in which the application scenario of the interface control method in the following description is control on a head unit interface. In this case, the interface control method 500 is executed by the head unit. However, it should be understood that the interface control method in this embodiment of the present application may also be applied to multiple other interface scenarios. In this case, the interface control method 500 is executed by another intelligent terminal.

[0053] S501: A user's voice command and the user's sound source position are acquired.

[0054] Specifically, the user's sound source position may be obtained using a sound source positioning method. For example, the microphone in a specific direction where the current sound source is located may be determined comprehensively based on the energy (strong / weak) of the sound signal collected by multiple microphones in different directions in the cockpit (e.g., microphones in the front left, front right, rear left, and rear right directions) and the preprocessed sound source positioning algorithm. In another example, a microphone array technology is used. Multiple microphones are arranged according to a specific rule. Multiple microphones synchronously collect sound signals, and the position where the noise sound source signal is transmitted is obtained based on the signal phase difference between the multiple microphones. In another example, a sound field test technology of an acoustic intensity probe is used. The acoustic intensity probe is a combination of two microphones arranged opposite to each other, which measures the sound pressure at multiple points in space and the magnitude and direction of the vibration velocity of the sound wave, and performs a scan test on the surface or near the envelope of the measurement object to obtain the sound field distribution situation near the measurement object, and then obtains the sound source position and sound source distribution.

[0055] S502: Information about the user's line of sight is acquired.

[0056] Specifically, the gaze information of the user may be obtained using an eye tracking method. For example, some eye structures and features whose relative positions do not change when the user's eyeball rotates may be used as a reference. Gaze change parameters are extracted between the position change features and the invariant features, and then the gaze direction is obtained using a geometric model or a mapping model to determine the position of the target window where the user's gaze is located. Optionally, the user is deemed to have the intention to control an interface (window) only if the user's gaze stays on the interface (window) for a period longer than a preset threshold. This reduces the false determination of the target window that occurs when the user's gaze stays on the interface window for a short period even though the user does not intend to control it.

[0057] S503: A target window on the interface is determined based on the sound source position and line-of-sight information.

[0058] As shown in FIG. 4, a target window on an interface may be determined based on a user's sound source position and the user's gaze information. The target window is an interface (or window) to be controlled by the user. In this case, the target window is located closest to the user's sound source position and in the gaze direction indicated by the user's gaze information. In other words, in this case, the window closest to the user's sound source position is the same as the window seen by the user's gaze.

[0059] However, in some cases, the window closest to the user's sound source position may be different from the window seen by the user's gaze. The window closest to the user's sound source position is denoted as the first window, and the window located in the gaze direction indicated by the user's gaze information is denoted as the second window. The first window is different from the second window. In this case, the target window is determined based on the priority of the sound source position and the priority of the gaze information. If the sound source position takes precedence over the gaze information, the first window is used as the target window. If the gaze information takes precedence over the sound source position, the second window is used as the target window. The priority of the sound source position and the priority of the gaze information are preset. For example, the sound source position may be set to take precedence over the gaze information at the time of factory setting, taking into account the user's habits or based on the user's statistical history data. Here, the priority of the sound source position over the gaze information or the priority of the gaze information over the sound source position may be reflected by the priority of the sound source position and the priority of the gaze information. Optionally, the user may adjust the priority of the sound source position and the priority of the gaze information based on requirements.

[0060] S504: The target window is controlled based on a voice command.

[0061] Specifically, a user's control intention is acquired based on the user's voice command, and then the user's control intention is executed in a target window based on the acquired application control information, for example, tapping, flicking, turning a page, etc., of the control in Figure 3.

[0062] Optionally, when the user's control intention is not executed in the target window, a self-learning optimization is performed, for example, the priority of the sound source position and the priority of the gaze information are adjusted based on the execution result of the voice command. For example, the head unit system may further include a self-learning module. The self-learning module adjusts the priority of the sound source position and the priority of the gaze information based on the execution result of the voice command. The priority may be determined and adjusted by setting and adjusting the priority information. For example, the sound source position has first priority information, and the gaze information has second priority information. The first priority information is used to determine the priority of the sound source position, and may be, for example, the weight or reliability level of the sound source position. The second priority information is used to determine the priority of the gaze information, and may be, for example, the weight or reliability of the gaze information.

[0063] Furthermore, there is a first correlation between the sound source position and the service indicated by the user's voice command, and there is a second correlation between the gaze information and the service indicated by the user's voice command. The priority information of the sound source position and the priority information of the gaze information may be a correlation or may be unrelated to the correlation. In this case, the priority of the sound source position and the priority of the gaze information may be determined based on the priority information and the correlation. Therefore, the method in this embodiment of the present application further includes a step of adjusting the first priority information and the second priority information based on the execution result of the user's voice command in the target window, and / or a step of adjusting the first correlation and the second correlation based on the execution result of the user's voice command in the target window. For example, the first priority information preset for the sound source position is 60%, and the second priority information preset for the gaze information is 40%. In this case, the first priority information is higher than the second priority information. Therefore, the sound source position takes precedence over the gaze information. If the window closest to the user's sound source position is different from the window seen by the user's gaze, the window closest to the user's sound source position is used as the target window. However, in this case, if the voice command is not executed in the window closest to the user's sound source position, the first priority information is appropriately decreased. For example, the first priority information is decreased to 58%, and the second priority information is appropriately increased. For example, the second priority information is increased to 42%. However, in this case, the first priority information is still higher than the second priority information. If the window closest to the user's sound source position is different from the window seen by the user's gaze, the window closest to the user's sound source position is still used as the target window. If the voice command fails multiple times in the window closest to the user's sound source position, the first priority information and the second priority information may be adjusted multiple times. If the first priority information is decreased to 48% and the second priority information is increased to 52%, the second priority information is higher than the first priority information. Therefore, the gaze information takes precedence over the sound source position.If the window closest to the user's sound source position is different from the window seen by the user's gaze, the window seen by the user's gaze is used as the target window. Furthermore, the sound source position and gaze information are further related to the service indicated by the user's voice command. Specifically, there is a first correlation between the sound source position and the service indicated by the user's voice command, and there is a second correlation between the gaze information and the service indicated by the user's voice command. For example, if the user's voice command is "play a movie", then based on common habits, the movie should be displayed in the gaze direction indicated by the user's gaze information. Thus, for the "play a movie" service, the first correlation may be preset to 20% and the second correlation may be preset to 80%. This is similar to the adjustment of the first priority information and the second priority information. If the voice command "play a movie" is not executed in the window of the gaze direction indicated by the user's gaze information, the first correlation is appropriately increased and the second correlation is appropriately decreased. If the first correlation is higher than the second correlation after the adjustment is performed multiple times, the user's voice command "play movie" is executed in the window closest to the user's sound source position. The adjustment of the first priority information, the second priority information, the first correlation, and the second correlation may adapt to the user's control habits to improve the user experience.

[0064] It should be understood that the specific values ​​of the first priority information, the second priority information, the first correlation, and the second correlation are merely examples, and each may instead be a decimal or integer number, or in another format.

[0065] In some cases, for example, on a music selection interface, some music names are too long, some music names contain symbols, foreign words, etc. As a result, it is difficult for users to express the names using voice. Moreover, if a user pays too much attention to text recognition on the interface, driving safety will be reduced. Therefore, the method in this embodiment of the present application further includes a step of displaying an icon in a target window, the icon including one or more indexes, and the user's voice command indicates a target index, and the target index belongs to one of the one or more indexes. FIG. 6 is a schematic diagram of two possible icons. It should be understood that FIG. 6 is only an example of the form of a possible icon index in this embodiment of the present application, and does not constitute a limitation to the present application. The interface in (a) in the figure includes three icons. Each icon indicates one TV series, and the name of the TV series may be displayed under each icon, and each icon includes one index. For example, the index of the first icon is (1), the index of the second icon is (2), and the index of the third icon is (3). In a real application, a user does not need to say the name of a TV series on the interface to open an icon, but only needs to say the index of the icon of the TV series to be opened. The interface in (b) in the figure includes one icon. The icon represents a TV series, and the icon includes 12 indexes, each index represents an episode of the TV series. In a real application, a user only needs to say the corresponding index to open an episode that the user wants to watch. The interface in (c) in the figure includes one icon. The icon represents a search result of songs, and includes three indexes. The indexes are presented in a list format, and each index represents a search result of songs "Music AB". Each search result corresponds to a song with the same name but different singers or albums. The song that the user wants to open may be opened when the user says the corresponding index.According to the method in this embodiment of the present application, after the user's voice command is obtained, an icon indicated by the user's voice command may be displayed in the target window. For example, only after the user's voice command is obtained, the corresponding interface is opened. The interface includes an icon corresponding to the service indicated by the user's voice command, and the display manner of the icon is one or more indexes. Alternatively, according to the method in this embodiment of the present application, the icon may be displayed in the target window. For example, one or more icons are originally displayed in the target window, and each icon includes one or more indexes. The user sends a voice command based on one or more indexes of each icon in the target window, and the voice command indicates the target index, and the target index belongs to one or more indexes in the target window. Then, an operation is performed on the target index based on the voice command. Since the index of the icon is displayed in the target window, the user can perform more convenient operations on the interface. The index is more simple and eye-catching than the specific name of the icon, so the user does not need to pay too much attention to it.

[0066] It should be understood that in the method provided in this embodiment of the present application, the icons on the interface are originally presented in the form of an index. Alternatively, a switch may be set to the index, and the icons on the interface may be presented in the form of an index only after the user turns on the switch. Alternatively, when the user controls the interface using a voice command, the icons on the interface are set to be presented in the form of an index, or when the user controls the interface manually, etc., the icons on the interface are not presented in the form of an index.

[0067] In addition to the form of index shown in FIG. 6, the method in this embodiment of the present application may also refer to the knowledge graph to attach a related label to the icon. Thus, the form of index further includes a label. For example, the icon of movie xxx (movie name) may be added with the label "movie ticket owed to xxx (movie director name)". The user may control the movie control when saying the content of the label.

[0068] Existing voice interface control technology cannot detect the content of the interface currently controlled by the user. For example, the interface currently controlled by the user is a movie selection interface, and the movie is on the movie selection interface. The name of the movie may be understood by the intelligent terminal as a voice control command with another function, resulting in malfunction. For example, the name of the movie includes the word "cold". For example, "I'm so cold". If the user directly says the name of the movie, the intelligent terminal may identify that the user is currently feeling cold based on the voice interface control technology, and then the air conditioner is switched to heating mode. According to this identification method, information about the interface currently controlled by the user can be obtained to realize the see-and-speak voice control effect. In addition, a multi-interface (window) mode may be supported to further improve the user experience. See-and-speak is also called "what you see is what you can speak" or "what you see is what you can speak". This means that the user can control the interface using voice commands within the screen range visible to the user without manual touch by the user or additional corpus learning costs.

[0069] In another implementation, in order to provide a better user experience, the method in this embodiment of the present application further includes setting the operation of non-entertainment functions to be prioritized over the operation of entertainment functions to ensure the user's basic requirements first. Entertainment functions include playing movies or TV series, playing songs, etc. Non-entertainment functions include controlling hardware inside the head unit, making calls on the head unit interface, sending messages, navigation, etc. In the above scenario, the air conditioner may be turned on preferentially. As another example, when a user sends a voice command of "Go Home", navigation to the user's residence and corresponding road condition information should be preferentially displayed instead of playing the song "Go Home".

[0070] Optionally, the method in this embodiment of the present application further comprises the step of setting some services to be preferentially displayed on a specific window when there are multiple windows or even multiple display screens. For example, if a vehicle includes multiple display screens, one of the display screens is located in front of a baby seat in the back seat, and the user's voice command is "play a children's program", the voice command can be set to directly act on the display screen in front of the baby seat. In other words, the children's program is preferentially displayed on the display screen in front of the baby seat. For the adjustment of priority, please refer to the above description. For the sake of brevity, the details will not be described again here.

[0071] In the following, with reference to the accompanying drawings, a head unit is used as an example to describe some applications of the interface control method in the embodiment of the present application. It should be understood that the interface control method in this embodiment of the present application can be applied to a scenario in which the same physical screen is divided into multiple screens, or a scenario in which an intelligent terminal includes multiple physical screens, such as the display screen 101, the central display screen 102, the display screen 103, the display screen 104, and the display screen 105 shown in FIG. 1, or a scenario in which there are multiple windows on the screen. The screen or window to be controlled by the user is determined with reference to the user's semantic information, gaze information, and / or sound source position. For example, the user sends a voice command of "navigate". The navigation interface may be displayed on the screen or window that the user wants to display based on the subjective intention indicated by the user's semantics. Alternatively, the target screen or target window to be controlled by the user may be determined based on the user's gaze information and / or sound source position, and then the navigation interface is displayed on the target screen or target window. It should be understood that the above example is only one example of the interface control method in this embodiment of the present application, and does not constitute a limitation to this embodiment of the present application.

[0072] FIG. 7 is a schematic diagram of the control of the head unit interface by the user in the driver's seat. As shown in FIG. 7, the head unit interface includes a driver screen and a passenger screen. When the user in the driver's seat sends the voice command of "open music", the user does not subjectively specify whether the voice command acts on the driver screen or the passenger screen, so the interface on which the operation is performed by the user may be further determined with reference to the user's sound source position and gaze information. Specifically, the user's sound source position and gaze information are obtained. Since the voice command is sent by the user in the driver's seat, the screen closest to the user's sound source position is the driver screen. Assuming that the user in the driver's seat is also looking at the driver screen, it is determined that the interface on which the operation is performed by the user is the driver screen. In this case, the operation is performed on the driver screen based on the user's voice command. In other words, the music app on the driver screen opens. The driver screen in FIG. 7 shows three icons indicating three different songs. It can be seen from FIG. 7 that the name of each song is small in text. As a result, it is difficult for the user to quickly recognize the name. In addition, if the user in the driver's seat is distracted for a long time during the driving process, it may cause danger. Therefore, according to the interface control method in this embodiment of the present application, the index of the icon on the interface is displayed, for example, icon (1) (icon 1 is blocked by the steering wheel), icon (2), and icon (3) on the driver screen in FIG. 7. Icons (1), (2) and (3) are simpler and more eye-catching than text, and are convenient for users to recognize. For example, in a real application, the user may ask "What does icon (2) show?" In this case, the voice assistant may answer the name of the song shown by icon (2) and the user may select it.

[0073] It should be understood that in practical application, the head unit screen may alternatively be a full screen split screen or a split window. For the operation method, please refer to the above description of Figure 7. In this embodiment of the present application, the details will not be described again here.

[0074] FIG. 8 is a schematic diagram of the control of the head unit interface by a passenger seat user. As shown in FIG. 8, the head unit interface includes a driver screen and a passenger seat passenger screen. When the passenger seat user sends the voice command of "open music", the user does not subjectively specify whether the voice command acts on the driver screen or the passenger seat passenger screen, so the interface on which the operation is performed by the user may be further determined with reference to the user's sound source position and gaze information. Specifically, the user's sound source position and gaze information are obtained. Since the voice command is sent by the passenger seat user, the screen closest to the user's sound source position is the passenger seat passenger screen. At this time, if the user is looking at the driver screen, the interface determined based on the user's gaze information is the driver screen. Therefore, the interface determined based on the sound source position does not match the interface determined based on the gaze information. In this case, it is necessary to further determine the interface based on the priority of the sound source position and the priority of the gaze information. Assuming that the sound source position takes precedence over the gaze information with a preset priority, the determined target interface is the passenger screen, and the interface on which the operation is performed by the passenger user is the driver screen. In this case, the user's voice command is not executed on the passenger screen, or the user's voice command may be executed on the passenger screen, but the user will send a voice of "execution error". Therefore, it is determined that the execution is unsuccessful, and then the corresponding priority information is adjusted. For the adjustment of the priority information, please refer to the above description of S504. In this embodiment of the present application, the details will not be described again here. After the priority information is adjusted one or more times, for the passenger user, the gaze information takes precedence over the sound source position. In this case, when the passenger user sends a voice command again and the interface determined based on the sound source position and the interface determined based on the gaze information do not match, the priority of the passenger user's gaze information is increased.

[0075] FIG. 9 is a schematic diagram of the control of the rear row head unit interface by the driver's seat user. As shown in FIG. 9, the vehicle includes multiple screens. The rear row head unit interface may be controlled by the driver's seat user or by the rear seat user. Assuming that the person who often sits in the right rear seat is a child, the driver's seat user may set the head unit interface in front of the right rear seat to display animation preferentially. For example, the driver's seat user may send a voice command of "play animation AB". In this case, the voice command is sent by the driver's seat user, and the animation AB is played on the head unit interface in front of the right rear seat based on the preset, although the driver's seat user cannot see the head unit interface in front of the right rear seat. Specifically, the multi-mode information may further include service information. A service (or application) corresponding to the service information is associated with or corresponds to the interface and is preferentially displayed on the interface.

[0076] 7 to 9 show the application of the interface control method according to this embodiment of the present application in multiple split screens. The interface control method in this embodiment of the present application may alternatively be applied to the scenario of an interface with multiple windows. FIG. 10 is a schematic diagram of simultaneously displaying three windows on one screen. As shown in FIG. 10, the upper left window of the screen displays multiple TV series icons, the upper right window displays a specific episode of the TV series, and the lower window displays a music search icon. In practical applications, the window closest to the user's sound source position may be determined as the target window, or the window in the user's line of sight may be determined as the target window. Specifically, the target window may be determined based on the priority of the sound source position and the priority of the line of sight information. For details, please refer to the above description of step 504 in FIG. 5. In this embodiment of the present application, details will not be described again here.

[0077] In conclusion, according to the interface control method in this embodiment of the present application, the target window to be controlled by the user is determined using at least two kinds of multi-modal information, such as the user's voice semantics, the user's sound source position, the user's gaze information, and service information. Then, the target window is controlled based on the voice command, so that the user can accurately control the target interface on the multiple head unit interfaces to improve the user experience. According to the interface control method in this embodiment of the present application, it may be further considered that if the target window determined separately based on the user's semantics, sound source position, and gaze information does not match, the target window can be determined based on the priority of the semantics, sound source position, and the user's gaze information. This can avoid the case where the user's voice command cannot be executed due to the target window determined separately based on the user's semantics, sound source position, and gaze information not matching. Furthermore, the priority of the user's semantics, sound source position, and gaze information can be adjusted based on the execution result to better adapt to the user's habits. The interface control method in this embodiment of the present application further includes a step of displaying an index of an icon in the target window. Icon indexes are simpler and more eye-catching than concrete text information, making them easier for users to recognize and select, reducing user distraction and improving driving safety.

[0078] 11 is a schematic flowchart of a user using voice commands to control an interface according to an embodiment of the present application. The scenario in FIG. 11 takes a head unit scenario as an example. When the head unit interface is split into screens or has multiple interfaces, a user using voice commands to control the head unit interface includes the following steps:

[0079] S1101: Send a voice command. A user sends a voice command to the voice signal processing module. The voice command includes a wake-up command and a task command, and may be, for example, "open xx (name of the voice assistant, which may be used as a wake-up command), music (which may be used as a task command)". The wake-up command and the task command may be said simultaneously using a complete command, or may not be said simultaneously using independent commands.

[0080] S1102: Perform voice recognition. After receiving the voice command, the voice signal processing module (also called voice service module) in the head unit performs semantic recognition on the user's voice. The semantic recognition may be any existing feasible semantic recognition technology or may be any semantic recognition technology developed in the future. This is not limited in this embodiment of the present application.

[0081] S1103: Send the voice recognition result The voice signal processing module then sends the semantic recognition result to the decision module.

[0082] S1104: Determine a target window based on the sound source position and gaze information. The determination module determines a target window (or interface) executed by the user based on the acquired sound source position and gaze information. The sound source position is the sound source position of the voice command, and the gaze information is the gaze information of the user who sent the voice command. Please refer to the above description of FIG. 5 for the specific process. The details will not be described again here. It should be understood that the sound source position and gaze information here can be acquired by sensors such as microphones and cameras not shown in FIG. 11.

[0083] S1105: obtain control information of the application in the target window. After determining the target window, the determination module obtains the control information of the application in the target window. Optionally, the determination module can send a command to the application in the target window, and the application in the target window can execute S1106, in other words, can send control information to the determination module.

[0084] S1107: Perform intention matching based on the speech recognition result: The determination module performs intention matching based on the semantic recognition result and the control information of the application in the target window to determine the user's intention.

[0085] S1108: Execute the corresponding action on the target control. Then, the decision module executes the corresponding action on the target control in the target window based on the user's intention, for example, tapping the corresponding control. In this way, when there are multiple interfaces (for example, split screen or multiple windows), the user can precisely control the application on the interface by using voice commands.

[0086] Optionally, the user may alternatively send a wake-up command first, such as "xx" (the name of the voice assistant that can be used as a wake-up command). The determination module first determines a target window based on the user's wake-up command by referring to the user's sound source position and gaze information, and then obtains the control information of the application in the target window. Then, the voice signal processing module performs semantic recognition by referring to the task command sent by the user, such as "open music", and sends the semantic recognition result to the determination module. The determination module performs intention matching based on the semantic recognition result and the control information of the application in the target window to determine the user's intention, and then performs a corresponding operation on the target control in the target window based on the user's intention.

[0087] Optionally, the method of Fig. 11 further includes, after the control information of the application in the target window is obtained, generating an index of an icon in the target window according to the control information. In this way, a user can control the target window according to the index of the icon.

[0088] 12 is a schematic flow chart of another interface control method according to an embodiment of the present application. As shown in FIG. 12, the method includes steps 1201 to 1204. For ease of explanation, in the following description, the control on the head unit interface is still used as an example in the application scenario of the interface control method. However, it should be understood that the interface control method in this embodiment of the present application may alternatively be applied to another multi-interface scenario.

[0089] S1201: A first voice command of a user is obtained, and the first voice command indicates a first service.

[0090] Specifically, a user sends a voice command to a voice assistant or the like in a head unit. The voice command indicates a certain type of service or application. In other words, the first service includes a service type or a service application. Indicating the first service may be understood as indicating the first service type or the first application. Here, for ease of explanation, an example in which the service is a music service is used. Of course, the service may alternatively be another type of service. The type of service is not limited in this embodiment of the present application. Furthermore, the voice command may directly indicate the service. For example, the voice command is "music" or the name of a music application (APP). Alternatively, the voice command may indirectly indicate the service. For example, the voice command is an operation on the service to indirectly determine the service. Similarly, the music service is used as an example. The voice command may be "open music" or "open xx (name of music application)".

[0091] S1202: An icon of a first service is displayed in a target window (or interface), the icon including a plurality of indexes.

[0092] An application corresponding to the first service is opened in a target window to be controlled by the user based on the user's voice command. If the first service is a service type, the target window may include multiple applications. In this case, the application with the highest priority may be opened, or an application running in the background may be opened. The application with the highest priority may be the application most frequently used by the user, or the application with the highest priority set by the user. Alternatively, multiple applications are all opened and selected by the user using an index. If the first service is a specific application, the application may be opened directly. Similarly, a music application is used as an example. The head unit displays a music icon in the target window based on the voice command. The icon includes multiple indexes. Each index may indicate a song, or each index may indicate an action on a song, such as play, add to favorites, or next song.

[0093] S1203: Obtain a second voice command of the user, where the second voice command indicates a target index, and the target index belongs to a plurality of indexes.

[0094] Since the object window provides the user with an index selection method, the requirements for the user's voice command are simplified, so that the voice command is simpler and has high recognition accuracy. Therefore, the user may select an index that matches the user's intention from multiple indexes and send a voice command corresponding to the index, for example, a simple number or a number voice command.

[0095] For example, the user again sends a voice command based on an index in the object window, the voice command indicating one of the indexes selected by the user in the object window, for example, the user selects an index of a song and selects an action on the song as adding to favorites.

[0096] S1204: A task which is of the first service and corresponds to the target index is executed.

[0097] After obtaining the voice command indicating the target index, the head unit may perform an operation corresponding to the voice command, in other words, one or more tasks of the first service.

[0098] For example, if the target index indicates a song, the song is played. If the target index indicates that the action for the song is to add to favorites, the song is added to favorites.

[0099] Optionally, the target window is the window closest to the user or in the user's line of sight.

[0100] According to the interface control method in this embodiment of the present application, the user can control the target window to be controlled by using voice without manual touch, which is convenient and safe. This improves the user experience. The icon in the target window includes an index, and the user's voice indicates the target index, so that the user can control the icon in the target window. The index is simpler and more eye-catching than concrete text information, which makes it easier for the user to recognize and select, avoids the user from being too distracted, and improves the driving safety in the scenario of control on the head unit interface.

[0101] Above, the interface control method in the embodiment of the present application has been described in detail with reference to Figs. 1 to 12. Hereinafter, the interface control device provided in the embodiment of the present application will be described with reference to Figs. 13 and 14. It should be understood that the description of the device embodiment corresponds to the description of the method embodiment. Therefore, for contents that are not described in detail, please refer to the above-mentioned method embodiment. For the sake of brevity, the details will not be described again here.

[0102] 13 is a schematic block diagram of a control device according to an embodiment of the present application. The device 1300 may specifically be a chip, an in-vehicle device, a vehicle, etc. The device 1300 includes an acquisition unit 1310 and a processing unit 1320. The acquisition unit 1310 may implement a corresponding communication function, and the processing unit 1320 is configured to process data. The acquisition unit 1310 may also be referred to as a communication interface or a communication unit. Specifically, when the device 1300 is a vehicle shown in FIG. 1, the acquisition unit 1310 may be the detection system 120 of FIG. 1 or a unit configured to implement the function of the detection system 120, and the processing unit 1320 may be the computing platform 150 of FIG. 1 or a unit configured to implement the function of the computing platform 150.

[0103] Optionally, the device 1300 may further include a memory unit. The memory unit may be configured to store instructions and / or data. The processing unit 1320 may read the instructions and / or data in the memory unit such that the device performs the above-mentioned method embodiments.

[0104] The apparatus 1300 may be configured to perform operations in the aforementioned method embodiments. In particular, the acquisition unit 1310 is configured to perform acquisition-related operations in the aforementioned method embodiments, and the processing unit 1320 is configured to perform processing-related operations in the aforementioned method embodiments.

[0105] The apparatus 1300 may perform steps or procedures corresponding to the method embodiments in the embodiments of the present application. The apparatus 1300 may include units configured to perform the methods of Figures 5 and 12. In addition, the units in the apparatus 1300 and other operations and / or functions described above are used separately to perform the corresponding procedures in the method embodiments on the second node side of Figures 5 and 12.

[0106] When the apparatus 1300 is configured to perform the method 500 of FIG. 5, the transceiver unit 1310 may be configured to perform steps 501 and 502 of the method 500, and the processing unit 1320 may be configured to perform processing steps 503 and 504 of the method 500.

[0107] Specifically, the acquisition unit 1310 is configured to acquire a user's voice command and a user's sound source position. The acquisition unit 1310 is further configured to acquire a user's gaze information. The processing unit 1320 is configured to determine a target window on the interface based on the sound source position and the gaze information. The processing unit 1320 is further configured to control the target window based on the voice command.

[0108] In some possible implementations, the interest window is located closest to the sound source position and in the line of sight direction indicated by the line of sight information.

[0109] In some possible implementations, the window closest to the sound source position is the first window, and the window located in the line of sight direction indicated by the line of sight information is the second window. The processing unit 1320 is specifically configured to determine the target window based on the priority of the sound source position and the priority of the line of sight information. The priority is predefined. If the sound source position takes precedence over the line of sight information, the first window becomes the target window, or if the line of sight information takes precedence over the sound source position, the second window becomes the target window.

[0110] In some possible implementations, the sound source position has first priority information and the line of sight information has second priority information, the first priority information being used to determine the priority of the sound source position and the second priority information being used to determine the priority of the line of sight information.

[0111] In some possible implementations, there is a first correlation between the sound source location and the service indicated by the voice command, and there is a second correlation between the gaze information and the service indicated by the voice command.

[0112] In some possible implementations, the apparatus 1300 is further configured to obtain an execution result of the voice command, adjust the first priority information and the second priority information based on the execution result, and adjust the first correlation and the second correlation based on the execution result.

[0113] In some possible implementations, the voice command shows an icon. The processing unit 1320 is specifically configured to display the icon in the target window, the icon including one or more indexes.

[0114] In some possible implementations, the apparatus 1300 is further configured to display an icon in the object window, the icon including one or more indexes, and the voice instruction indicates the object index, the object index belonging to the one or more indexes.

[0115] When the apparatus 1300 is configured to perform the method 1200 of FIG. 12 , the transceiver unit 1310 may be configured to perform steps 1201 and 1203 of the method 1200, and the processing unit 1320 may be configured to perform processing steps 1202 and 1204 of the method 1200.

[0116] Specifically, the acquiring unit 1310 is configured to acquire a first voice command of the user, where the first voice command indicates a first service. The processing unit 1320 is configured to display an icon of the first service in a target window, where the icon includes a plurality of indexes. The acquiring unit 1310 is further configured to acquire a second voice command of the user, where the second voice command indicates a target index, where the target index belongs to the plurality of indexes. The processing unit 1320 is further configured to execute a task that is of the first service and corresponds to the target index.

[0117] In some possible implementations, the target window is the window closest to the user or the window in the user's line of sight.

[0118] It should be understood that the specific processes by which the units perform the above corresponding steps have been described in detail in the above method embodiments, and for the sake of brevity, the details will not be described here.

[0119] As shown in FIG. 14, an embodiment of the present application further provides an interface control device 1400. The computing device 1400 shown in FIG. 14 may include a memory 1410, a processor 1420, and a communication interface 1430. The memory 1410, the processor 1420, and the communication interface 1430 are connected via an internal connection path. The memory 1410 is configured to store instructions. The processor 1420 is configured to execute the instructions stored in the memory 1420 and control the communication interface 1430 to receive the input sample or send the prediction result. Optionally, the memory 1410 may be coupled to the processor 1420 via an interface or integrated into the processor 1420. The interface control device 1400 may be the vehicle 100 shown in FIG. 1. The communication interface 1430 may be the detection system 120 of FIG. 1 configured to perform step 501 and step 502 of FIG. 5. The processor 1420 may be the computing platform 150 of FIG. 1 configured to perform steps 503 and 504 of FIG.

[0120] In addition, the communication interface 1430 implements communication between the communication device 1400 and other devices or communication networks using a transceiver device such as, for example, but not limited to, a transceiver. The communication interface 1430 may further include an input / output interface.

[0121] In one embodiment process, the steps of the above-mentioned method may be implemented by using integrated logic circuits of hardware in the processor 1420 or by using instructions in the form of software. The method disclosed with reference to the embodiments of the present application may be directly implemented by a hardware processor, or may be implemented by using a combination of hardware and software modules in the processor. The software modules may be located in a storage medium mature in the art, for example, in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory 1410, and the processor 1420 reads the information of the memory 1410 and completes the steps of the above-mentioned method in combination with the hardware of the processor 1420. To avoid repetition, the details will not be described again here.

[0122] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), another general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or another programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0123] It should also be understood that in the embodiments of the present application, the memory may include read-only memory and random access memory to provide instructions and data to the processor. A part of the processor may further include a non-volatile random access memory. For example, the processor may further store device type information.

[0124] An embodiment of the present application further provides a chip. The chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to perform the method of FIG. 5 or FIG. 12.

[0125] An embodiment of the present application further provides a computing device, the computing device including a memory configured to store a computer program and a processor configured to call the computer program from the memory, the computing device being capable of executing the method of FIG.

[0126] As used herein, terms such as "component," "module," and "system" are used to indicate computer-related entities, hardware, firmware, a combination of hardware and software, software, or software being executed. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. As illustrated using the figures, both a computing device and an application running on a computing device may be a component. One or more components may reside within a process and / or thread of execution, and a component may be located on one computer and / or distributed across two or more computers. Additionally, these components may execute from various computer-readable media that store various data structures. For example, a component may communicate using local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component in a local system, a distributed system, and / or over a network such as the Internet that interacts with other systems using signals).

[0127] Those skilled in the art may recognize that the units and algorithm steps in the examples described with reference to the embodiments disclosed herein may be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but the implementation should not be considered to go beyond the scope of this application.

[0128] For the sake of convenience, the detailed operation processes of the above-mentioned systems, devices and units may be clearly understood by those skilled in the art by referring to the corresponding processes in the above-mentioned method embodiments, and the details will not be described again here.

[0129] In some embodiments provided in the present application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the described device embodiment is only an example. For example, the division into units is only a logical functional division, and may be other divisions in actual implementation. For example, multiple units or components may be combined or integrated into another system, and some features may be ignored or not implemented. In addition, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0130] The units described as separate parts may or may not be physically separated, and the parts shown as units may or may not be physical units, and may be located in one location or distributed over multiple network units. Some or all of the units may be selected based on the actual requirements to achieve the objectives of the solutions in the embodiments.

[0131] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, and each of the units may exist physically alone, or two or more units may be integrated into one unit.

[0132] When the functions are implemented in the form of a software functional unit and sold or used as an independent product, the functions may be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented in the form of a software product, or the part that contributes to the prior art, or all or part of the technical solutions. The computer software product includes some instructions stored in a storage medium to instruct a computer device (which may be a personal computer, a server, a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0133] The above description is merely a specific implementation of this application, and is not intended to limit the scope of protection of this application. Any variations or replacements that are easily conceived by those skilled in the art within the technical scope disclosed in this application shall be within the scope of protection of this application. Therefore, the scope of protection of this application shall be subject to the scope of protection of the claims. [Explanation of symbols]

[0134] 101 Digital Dashboard Display Screen 102 Central display screen 103 Display screen 104 Display screen 105 Display screen 106 Camera 1300 Interface Control Device 1310 Acquisition Department 1320 Processing section 1400 Interface Control Device 1410 Memory 1420 Processor 1430 Communication Interface

Claims

1. obtaining a voice command of a user and a sound source position of said user; acquiring gaze information of the user; determining a target window on an interface based on the sound source position and the line of sight information; controlling the target window based on the voice command; An interface control method comprising: a window closest to the sound source position is a first window, a window in a line of sight direction indicated by the line of sight information is a second window, and the step of determining a target window on an interface based on the sound source position and the line of sight information includes: A method comprising a step of determining the target window based on a priority of the sound source position and a priority of the line of sight information, wherein if the priority of the sound source position is higher than the priority of the line of sight information, the first window becomes the target window, or if the priority of the line of sight information is higher than the priority of the sound source position, the second window becomes the target window.

2. A step of obtaining a voice command of a user and a sound source position of the user; acquiring gaze information of the user; determining a target window on an interface based on the sound source position and the line of sight information; controlling the target window based on the voice command; An interface control method comprising: the target window is closest to the sound source position and is located in a line of sight direction indicated by the line of sight information; A method, wherein the sound source position has first priority information and the line of sight information has second priority information, the first priority information is used to determine the priority of the sound source position, and the second priority information is used to determine the priority of the line of sight information.

3. The method of claim 1 , wherein the priority of the sound source location and the priority of the gaze information are predefined.

4. 4. The method of claim 3, wherein the sound source position has first priority information and the line of sight information has second priority information, the first priority information being used to determine the priority of the sound source position and the second priority information being used to determine the priority of the line of sight information.

5. The method further comprising: obtaining a result of executing the voice command; The method of claim 4 , further comprising: adjusting the first priority information and the second priority information based on the execution result.

6. The method of claim 1 or any one of claims 3 to 5, wherein there is a first correlation between the sound source location and the service indicated by the voice command, and there is a second correlation between the gaze information and the service indicated by the voice command.

7. An acquisition unit configured to acquire a voice command of a user and a sound source position of the user, An acquisition unit, the acquisition unit being further configured to acquire gaze information of the user; a processing unit configured to determine a target window on an interface based on the sound source position and the line of sight information, a processing unit, the processing unit being further configured to control the target window based on the voice command; An interface control device comprising: a window closest to the sound source position is a first window, a window in a line of sight direction indicated by the line of sight information is a second window, and the processing unit An interface control device that is specifically configured to determine the target window based on the priority of the sound source position and the priority of the gaze information, and when the priority of the sound source position is higher than the priority of the gaze information, the first window becomes the target window, or when the priority of the gaze information is higher than the priority of the sound source position, the second window becomes the target window.

8. An acquisition unit configured to acquire a voice command of a user and a sound source position of the user, comprising: An acquisition unit, the acquisition unit being further configured to acquire gaze information of the user; a processing unit configured to determine a target window on an interface based on the sound source position and the line of sight information, a processing unit, the processing unit being further configured to control the target window based on the voice command; An interface control device comprising: the target window is closest to the sound source position and is located in a line of sight direction indicated by the line of sight information; An apparatus, wherein the sound source position has first priority information and the line of sight information has second priority information, the first priority information is used to determine the priority of the sound source position, and the second priority information is used to determine the priority of the line of sight information.

9. The apparatus of claim 7 , wherein the priority of the sound source location and the priority of the gaze information are predefined.

10. 10. The apparatus of claim 9, wherein the sound source position has first priority information and the line of sight information has second priority information, the first priority information being used to determine the priority of the sound source position and the second priority information being used to determine the priority of the line of sight information.

11. The apparatus, Obtaining a result of the execution of the voice command; The apparatus of claim 10 , further configured to adjust the first priority information and the second priority information based on the execution result.

12. The device of claim 7 , wherein there is a first correlation between the sound source location and a service indicated by the voice command, and there is a second correlation between the gaze information and the service indicated by the voice command.

13. A vehicle equipped with an apparatus as described in any one of claims 7 to 11.

14. A vehicle equipped with the device described in claim 12.

Images