Control triggering method, vehicle-mounted terminal and storage medium

Abstract

The invention provides a control triggering method, a vehicle-mounted terminal and a storage medium, and relates to the technical field of vehicles. When it is determined that at least two target controls associated with the keyword are included in the multiple controls, the orientation of a user sending the voice data is recognized; controlling a vehicle-mounted camera to shoot towards the direction of the user sending the voice data to obtain an image containing the eyes of the user; determining coordinates of fixation points of eyeballs of the user eyes on the human-computer interaction interface according to the image containing the user eyes; retrieving a target control within a preset distance range from the coordinates of the fixation point in the human-computer interaction interface; and under the condition that one target control is retrieved, the corresponding operation when the retrieved target control is triggered is executed, so that one target control meeting user requirements is automatically triggered under the condition that the target controls associated with the keywords in the voice data of the multiple users exist, the operation steps of the users are saved, and the user experience is improved. And the user experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of vehicles, and in particular, to a control triggering method, an in-vehicle terminal, and a storage medium. Background Art

[0002] With the development of technology, the intelligent cockpit system is making great strides towards increasing intelligence. Among them, the in-vehicle voice assistant is a convenient intelligent software, enabling the intelligent cockpit of the vehicle to have the function of voice visible and audible. That is to say, the vehicle can listen to the user's voice, extract the keywords in the voice data, and execute the functions associated with the keywords. For example, execute the page collection function of the application associated with the collection control corresponding to the keyword, execute the closing function of the page of the application associated with the closing control corresponding to the keyword, etc.

[0003] However, with the increasing size of in-vehicle screens and the growing popularity of triple screens, multiple application interfaces are displayed on multiple screens or multiple areas of a single screen in the vehicle. When a keyword is associated with multiple controls, it is necessary to prompt the user to manually trigger one of the associated controls, which is a cumbersome process and results in a low user experience. Summary of the Invention

[0004] The present application provides a control triggering method, an in-vehicle terminal, and a storage medium, which are used to solve the problem in the prior art that when a keyword in voice data is associated with multiple controls, it is necessary to prompt the user to manually trigger one of the associated controls, which is a cumbersome process and results in a low user experience.

[0005] The present application provides a control triggering method, which is applied to an in-vehicle terminal. The in-vehicle terminal displays a human-machine interaction interface, and the human-machine interaction interface includes multiple controls. The method provided by the present application includes:

[0006] Listen to the voice data issued by the user and identify the keywords in the voice data;

[0007] When it is determined that at least two target controls associated with the keyword are included in the multiple controls, identify the orientation of the user who issued the voice data;

[0008] Control the in-vehicle camera to take a picture in the direction of the user who issued the voice data, and obtain an image including the user's eyes;

[0009] According to the image including the user's eyes, determine the coordinates of the fixation point of the user's eyes on the human-machine interaction interface;

[0010] Retrieve the target control within the preset distance range from the coordinates of the fixation point in the human-machine interaction interface;

[0011] When one target control is retrieved, execute the operation corresponding to the triggered retrieved target control.

[0012] In some embodiments, the human - machine interaction interface includes display windows of multiple applications. Each display window of an application includes at least one control, and at least one keyword is associated with at least two controls in the display windows of multiple applications.

[0013] In some embodiments, determining at least two target controls associated with a keyword among multiple controls includes:

[0014] Searching for a function label corresponding to the keyword from a preset first mapping relation table, where the first mapping relation includes the corresponding relations between various different keywords and their corresponding function labels;

[0015] Searching for at least two target control IDs corresponding to the function label from a preset second mapping relation table, where the second mapping relation includes the corresponding relations between various different function labels and their corresponding control IDs.

[0016] In some embodiments, the vehicle - mounted terminal includes a plurality of microphones arranged in an array. Identifying the orientation of a user who emits voice data includes:

[0017] Identifying the orientation of a user who emits voice data according to the geometric layout of the preset microphones, the phase difference and time difference of the voice data arriving at each microphone, and a preset sound - source localization algorithm.

[0018] In some embodiments, determining the coordinates of the fixation point of the eyeball on the human - machine interaction interface according to an image of the user's eyes includes:

[0019] Determining the fixation direction and coordinates of the eyeball from the image of the user's eyes according to an eyeball - tracking algorithm;

[0020] Determining the coordinates of the fixation point of the eyeball on the human - machine interaction interface according to the fixation direction of the eyeball, the coordinates of the eyeball, and a preset mapping model.

[0021] In some embodiments, determining the coordinates of the fixation point of the eyeball on the human - machine interaction interface according to the fixation direction of the eyeball, the coordinates of the eyeball, and a preset mapping model includes:

[0022] The mapping model is a pre - trained neural network model. Inputting the fixation direction of the eyeball and the coordinates of the eyeball into the neural network model to determine the coordinates of the fixation point of the eyeball on the human - machine interaction interface, where the neural network model is obtained by training multiple training samples input into a network to be trained, and each training sample includes the fixation direction of a historical eyeball, the coordinates of the historical eyeball, and the historical coordinates of the fixation point of the eyeball on the human - machine interaction interface corresponding thereto;

[0023] The mapping model is the third mapping relation table. According to the gaze direction of the eyeball and the coordinates of the eyeball, the coordinates of the fixation point of the eyeball on the human-computer interaction interface are found from the third mapping relation table, where the coordinates of the fixation point of the eyeball on the human-computer interaction interface corresponding to the gaze direction and the coordinates of each group of eyeballs in the third mapping relation table.

[0024] In some embodiments, after retrieving the target controls within the preset distance range from the coordinates of the fixation point in the human-computer interaction interface, the method provided by the present application further includes:

[0025] In the case of retrieving multiple target controls, output a prompt message for indicating control selection.

[0026] In a second aspect, the present application further provides a vehicle-mounted terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the vehicle-mounted terminal executes the method provided in the first aspect of the present application.

[0027] In a third aspect, the present application further provides a storage medium storing a computer program, and when the computer program is executed by a processor, the computer executes the method provided in the first aspect of the present application.

[0028] In a fourth aspect, the present application further provides a computer program product, including a computer program, and when the computer program is run, the vehicle-mounted terminal executes the method provided in the first aspect of the present application.

[0029] The present application provides a control triggering method, a vehicle-mounted terminal, and a storage medium, which can listen to the voice data sent by a user and identify the keywords in the voice data; in the case of determining that at least two target controls associated with the keywords are included in multiple controls, identify the orientation of the user who sent the voice data; control the vehicle-mounted camera to take a picture in the direction of the user who sent the voice data to obtain an image including the user's eyes; determine the coordinates of the fixation point of the eyeball of the user's eyes on the human-computer interaction interface according to the image including the user's eyes; retrieve the target controls within the preset distance range from the coordinates of the fixation point in the human-computer interaction interface; in the case of retrieving one target control, perform the operation corresponding to when the retrieved target control is triggered. In this way, when there are target controls associated with the keywords in the voice data of multiple users, one of the target controls that meets the user's needs is automatically triggered, saving the user's operation steps and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0031] Figure 1 It is a flowchart of the control trigger method provided by the embodiment of the present application;

[0032] Figure 2 It is a schematic diagram of the human-computer interaction interface provided by the embodiment of the present application;

[0033] Figure 3 It is a functional module block diagram of the control trigger device provided by the embodiment of the present application. Detailed implementation manners

[0034] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In addition, in the following description, the descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present disclosure.

[0035] Various schematic structural diagrams according to embodiments of the present disclosure are shown in the accompanying drawings. These figures are not drawn to scale, where for the purpose of clear expression, some details are enlarged and some details may be omitted. The shapes of various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary. In practice, there may be deviations due to manufacturing tolerances or technical limitations, and those skilled in the art can design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0036] In the context of the present disclosure, when a layer / element is referred to as being "on" another layer / element, the layer / element can be directly on the other layer / element, or there can be an intermediate layer / element between them. Additionally, if a layer / element is "on" another layer / element in one orientation, then when the orientation is reversed, the layer / element can be "under" the other layer / element.

[0037] Hereinafter, the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail with specific embodiments. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0038] Please refer to Figure 1, an embodiment of the present application provides a control triggering method, which is applied to an in-vehicle terminal. The method provided by the embodiment of the present application includes:

[0039] S101: The in-vehicle terminal displays a human-computer interaction interface 201, and the human-computer interaction interface 201 includes multiple controls.

[0040] In some embodiments, the human-computer interaction interface 201 includes display windows of multiple applications. Each display window of an application includes at least one control, and at least one keyword is associated with at least two controls in the display windows of multiple applications. For example, as Figure 2 shown, there are display windows of multiple applications floating on the human-computer interaction interface 201. For example, there is a display window 202 of a short video application, a display window 203 of a music application, etc. The multiple controls include a favorite control A, a comment control, a follow control, etc. in the display window 202 of the short video application, and a favorite control B, a forward control, a play control, etc. in the display window 203 of the music application, which are not limited here.

[0041] S102: Listen to the voice data sent by the user and identify the keyword in the voice data.

[0042] Exemplarily, an array microphone of the in-vehicle terminal can be used to listen to the voice data sent by the user. For example, if the voice data is "Hello, please favorite the data on the current page", the keyword "favorite" can be extracted from the voice data. Among them, by performing word segmentation on the voice data, a vocabulary belonging to a preset keyword set can be used to extract keywords such as "favorite".

[0043] S103: In the case of determining that at least two target controls associated with the keyword are included in the multiple controls, identify the orientation of the user who sent the voice data.

[0044] Exemplarily, the function label corresponding to the keyword can be found from a preset first mapping relationship table, where the first mapping relationship includes the corresponding relationship between each different keyword and the corresponding function label; from a preset second mapping relationship table, at least two target control IDs corresponding to the function label can be found, where the second mapping relationship includes the corresponding relationship between each different function label and the corresponding control ID. For example, as Figure 2 shown, at least two target controls associated with the keyword "favorite" are the favorite control A in the display window 202 of the short video application and the favorite control B in the display window 203 of the music application.

[0045] In addition, the in-vehicle terminal includes a plurality of microphones arranged in an array, and can identify the orientation of the user who emits the voice data according to the preset geometric layout of the microphones, the phase difference and time difference of the voice data reaching each microphone, and the preset sound source localization algorithm (such as beamforming algorithm, or minimum mean square error algorithm, etc.).

[0046] S104: Control the in-vehicle camera to take a picture in the direction of the user who emits the voice data, and obtain an image including the user's eyes.

[0047] For example, when the orientation of the user who emits the voice data is at the driver's seat, control the in-vehicle camera to take a picture at the driver's seat; when the orientation of the user who emits the voice data is at the passenger seat, control the in-vehicle camera to take a picture at the passenger seat.

[0048] S105: Determine the coordinates of the fixation point of the user's eyes on the human-machine interaction interface 201 according to the image including the user's eyes.

[0049] Specifically, the gaze direction and coordinates of the eyeball can be determined from the image of the user's eyes according to the eye tracking algorithm; according to the gaze direction of the eyeball, the coordinates of the eyeball, and the preset mapping model, the coordinates of the fixation point of the eyeball on the human-machine interaction interface 201 are determined.

[0050] In some embodiments, the mapping model is a pre-trained neural network model. The gaze direction and coordinates of the eyeball are input into the neural network model to determine the coordinates of the fixation point of the eyeball on the human-machine interaction interface 201. Among them, the neural network model is obtained by training multiple training samples input into the network to be trained, and each training sample includes the gaze direction of the historical eyeball, the coordinates of the historical eyeball, and the historical coordinates of the fixation point of the eyeball on the human-machine interaction interface.

[0051] In some other embodiments, the mapping model is a third mapping relation table. According to the gaze direction and coordinates of the eyeball, the coordinates of the fixation point of the eyeball on the human-machine interaction interface 201 are found from the third mapping relation table. Among them, the third mapping relation table includes the coordinates of the fixation point of the eyeball corresponding to the gaze direction and coordinates of each group of eyeballs on the human-machine interaction interface.

[0052] S106: Retrieve the target control within the human-machine interaction interface 201 whose coordinates are within a preset distance range from the fixation point.

[0053] Exemplarily, the preset distance range can be, but is not limited to, within 5 cm. When the target space is a favorite control, still as Figure 2 shown, the target control within 5 cm of the coordinates a of the fixation point is the favorite control A within the display window 202 of the short video application.

[0054] S107: When a target control is retrieved, perform the operation corresponding to the triggering of the retrieved target control.

[0055] For example, when the target control is the favorite control A in the display window 202 of the short video application, perform the operation of favoriting the short video currently being played by the short video application.

[0056] In addition, when multiple target controls are retrieved, output a prompt message for indicating control selection for the user to select a control that meets personal needs for triggering.

[0057] In summary, the embodiment of the present application provides a control triggering method, which can listen to the voice data sent by the user and identify the keywords in the voice data; when it is determined that at least two target controls associated with the keywords are included in multiple controls, identify the orientation of the user who sent the voice data; control the in-vehicle camera to take a picture in the direction of the user who sent the voice data to obtain an image including the user's eyes; determine the coordinates of the fixation point of the user's eyes on the human-computer interaction interface 201 according to the image including the user's eyes; retrieve the target control within the preset distance range from the coordinates of the fixation point in the human-computer interaction interface 201; when a target control is retrieved, perform the operation corresponding to the triggering of the retrieved target control. In this way, when there are target controls associated with the keywords in the voice data of multiple users, one of the target controls that meets the user's needs is automatically triggered, saving the user's operation steps and improving the user experience.

[0058] As Figure 3 shown, the embodiment of the present application also provides a control triggering device configured in the vehicle terminal. It should be noted that for the control triggering device provided by the embodiment of the present application, its basic principle and the technical effects generated are the same as those of the above embodiment. For the sake of brief description, for the parts not mentioned in the embodiment of the present application, reference may be made to the corresponding content in the above embodiment. The control triggering device provided by the embodiment of the present application includes an interface display unit, a voice listening unit, a sound source localization unit, an image capturing unit, a fixation point determination unit, a control retrieval unit, and a function execution unit, where

[0059] The interface display unit is used to display the human-computer interaction interface 201, and the human-computer interaction interface 201 includes multiple controls.

[0060] The voice listening unit is used to listen to the voice data sent by the user and identify the keywords in the voice data.

[0061] The sound source localization unit is used to identify the orientation of the user who sent the voice data when it is determined that at least two target controls associated with the keywords are included in multiple controls.

[0062] An image capturing unit for controlling the on-vehicle camera to capture in the direction of the user who emits voice data, so as to obtain an image including the user's eyes.

[0063] A fixation point determination unit for determining the coordinates of the fixation point of the user's eye ball on the human-computer interaction interface 201 according to the image including the user's eyes.

[0064] A control retrieval unit for retrieving a target control within the human-computer interaction interface 201 whose coordinates are within a preset distance range from the fixation point.

[0065] A function execution unit for, in the case of retrieving a target control, performing the operation corresponding to when the retrieved target control is triggered.

[0066] In some embodiments, the human-computer interaction interface 201 includes display windows of multiple application programs. Each display window of an application program includes at least one control, and at least one keyword is associated with at least two controls in the display windows of multiple application programs.

[0067] In some embodiments, the device provided in the embodiment of the present application may further include a control determination unit for looking up the function label corresponding to the keyword from a preset first mapping relationship table, where the first mapping relationship includes the corresponding relationship between each different keyword and the corresponding function label. Looking up at least two target control IDs corresponding to the function label from a preset second mapping relationship table, where the second mapping relationship includes the corresponding relationship between each different function label and the corresponding control ID.

[0068] In some embodiments, the on-vehicle terminal includes a plurality of microphones arranged in an array. The sound source localization unit is specifically configured to identify the direction of the user who emits voice data according to the preset geometric layout of the microphones, the phase difference and time difference of the voice data arriving at each microphone, and the preset sound source localization algorithm.

[0069] In some embodiments, the fixation point determination unit is specifically configured to determine the fixation direction and coordinates of the eye ball from the image of the user's eyes according to the eye tracking algorithm; and determine the coordinates of the fixation point of the eye ball on the human-computer interaction interface 201 according to the fixation direction of the eye ball, the coordinates of the eye ball, and the preset mapping model.

[0070] In some embodiments, the mapping model is a pre-trained neural network model. The fixation point determination unit is specifically configured to input the fixation direction of the eyeball and the coordinates of the eyeball into the neural network model to determine the coordinates of the fixation point of the eyeball on the human-computer interaction interface 201. The neural network model is obtained by training a plurality of training samples input into the network to be trained. Each training sample includes the fixation direction of the historical eyeball, the coordinates of the historical eyeball, and the historical coordinates of the corresponding fixation point of the eyeball on the human-computer interaction interface 201.

[0071] In other embodiments, the mapping model is a third mapping relation table. The fixation point determination unit is specifically configured to find the coordinates of the fixation point of the eyeball on the human-computer interaction interface 201 from the third mapping relation table according to the fixation direction of the eyeball and the coordinates of the eyeball. The third mapping relation table includes the coordinates of the fixation points of the eyeballs corresponding to the fixation directions and coordinates of each group of eyeballs on the human-computer interaction interface 201.

[0072] In some embodiments, the device provided by the embodiments of the present application further includes a prompt information output unit, configured to output prompt information for indicating control selection in the case of retrieving a plurality of target controls.

[0073] In addition, the embodiments of the present application further provide a vehicle-mounted terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the vehicle-mounted terminal executes the method provided by the above embodiments of the present application.

[0074] In addition, the embodiments of the present application further provide a storage medium storing a computer program. When the computer program is executed by a processor, the computer executes the method provided by the above embodiments of the present application.

[0075] In addition, the present application further provides a computer program product, including a computer program. When the computer program is run, the vehicle-mounted terminal executes the method provided by the above embodiments of the present application.

[0076] In the above description, technical details such as the composition of each layer are not described in detail. However, those skilled in the art should understand that various technical means can be used to form layers, regions, etc. of the required shapes. In addition, in order to form the same structure, those skilled in the art can also design methods that are not exactly the same as the methods described above. In addition, although the above embodiments are described separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination.

[0077] Although the preferred embodiments of the present application have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn of the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present application.

[0078] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A control triggering method, characterized in that: Applied to a vehicle-mounted terminal, the vehicle-mounted terminal displays a human-computer interaction interface, the human-computer interaction interface includes a plurality of controls, and the method includes: Monitor the voice data sent by the user and identify keywords in the voice data; In the case of determining that the plurality of controls include at least two target controls associated with the keyword, identifying the location of the user who issued the voice data; Controlling the vehicle-mounted camera to shoot in the direction of the user who sent the voice data to obtain an image including the user's eyes; Determining the coordinates of a gaze point of an eyeball of the user on the human-computer interaction interface according to the image containing the user's eyes; Retrieving a target control in the human-computer interaction interface whose coordinates are within a preset distance range from the gaze point; In the case where one of the target controls is retrieved, the corresponding operation when the retrieved target control is triggered is performed.

2. The method according to claim 1, characterized in that The human-computer interaction interface includes display windows of multiple application programs, each display window of the application program includes at least one control, and at least one keyword is associated with at least two controls in the display windows of the multiple application programs.

3. The method according to claim 1, characterized in that The determining that at least two target controls associated with the keyword are included in the multiple controls includes: Searching for a function tag corresponding to the keyword from a preset first mapping relationship table, wherein the first mapping relationship includes a correspondence between different keywords and corresponding function tags; At least two target control IDs corresponding to the function label are found from a preset second mapping relationship table, wherein the second mapping relationship includes a correspondence relationship between each different function label and the corresponding control ID.

4. The method according to claim 1, characterized in that: The vehicle-mounted terminal includes a plurality of microphones arranged in an array, and the identifying the position of the user who sends the voice data includes: The position of the user who sends the voice data is identified based on the preset geometric layout of the microphones, the phase difference and time difference of the voice data reaching each microphone, and the preset sound source localization algorithm.

5. The method according to claim 1, characterized in that The step of determining the coordinates of a gaze point of an eyeball of the user on the human-computer interaction interface according to the image containing the user's eyes comprises: determining the gaze direction and the coordinates of the eye from the image of the user's eye according to an eye tracking algorithm; The coordinates of the gaze point of the eyeball on the human-computer interaction interface are determined according to the gaze direction of the eyeball, the coordinates of the eyeball and a preset mapping model.

6. The method according to claim 5, characterized in that Determining the coordinates of the gaze point of the eyeball on the human-computer interaction interface according to the gaze direction of the eyeball, the coordinates of the eyeball and a preset mapping model, includes: The mapping model is a pre-trained neural network model, and the gaze direction of the eyeball and the coordinates of the eyeball are input into the neural network model to determine the coordinates of the gaze point of the eyeball on the human-computer interaction interface, wherein the neural network model is obtained by inputting a plurality of training samples into the network to be trained, and each of the training samples includes the historical gaze direction of the eyeball, the historical coordinates of the eyeball and the corresponding historical coordinates of the gaze point of the eyeball on the human-computer interaction interface; The mapping model is a third mapping relationship table. According to the gaze direction of the eyeball and the coordinates of the eyeball, the coordinates of the gaze point of the eyeball on the human-computer interaction interface are found from the third mapping relationship table, wherein the gaze directions of the eyeballs and the coordinates of the eyeballs in each group in the third mapping relationship table respectively correspond to the coordinates of the gaze point of the eyeball on the human-computer interaction interface.

7. The method according to any one of claims 1 to 6, characterized in that: After retrieving the target control in the human-computer interaction interface whose coordinates are within a preset distance range from the gaze point, the method further includes: When a plurality of target controls are retrieved, prompt information for indicating control selection is output.

8. A vehicle-mounted terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the vehicle-mounted terminal executes the method according to any one of claims 1 to 7.

9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the computer is caused to perform the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed, the vehicle-mounted terminal executes the method as claimed in any one of claims 1 to 7.

Images