A human-computer interaction method for intelligent cockpit based on sound source localization

Abstract

The present invention discloses a human-computer interaction method for an intelligent cockpit based on sound source positioning, wherein a plurality of sound capture devices and at least one host are arranged in the intelligent cockpit; the method further comprises a plurality of commands for human-computer interaction operations and a pre-set command list, wherein the command list comprises all the commands and the sound source position corresponding to each command; the method further comprises a sound positioning module that processes the sound and obtains the sound source position and a control module that compares the obtained sound source position with the command list to obtain the corresponding command. The beneficial effects of the present invention are as follows: different operation commands are corresponded by the position of the sound source, so as to realize multi-person or multi-point human-computer interaction; the method can also directly recognize commands through some phonemes by converting speech into phonemes, thereby reducing the process of semantic recognition, which not only simplifies the complexity of the recognition engine, but also greatly improves the recognition speed.

Description

Technical Field

[0001] The present invention relates to the field of intelligent vehicles, and in particular to an intelligent cockpit human-computer interaction method based on sound source localization. Background Art

[0002] With the booming development of electric vehicles, it has become increasingly common to configure large displays and high-performance hosts in small buses. However, the relevant peripheral software is still in its infancy. For example, there are very few games that are compatible with electric vehicle cockpits. Another important reason for this is that, due to the environment of the cockpit, input devices such as mice and handles are generally not configured, and the touch screen method also has high requirements for the user's position, and the gaming experience is not good. How to improve the convenience of human-computer interaction in the cockpit and improve the gaming experience is a technical problem that needs to be solved urgently. On the other hand, the method of realizing human-computer interaction through voice and controlling smart devices has become very mature, and there is also a trend of popularization in buses. For example, there are practical applications for waking up the in-vehicle navigation and even controlling smart devices such as smart air conditioners in the car through voice. At present, human-computer interaction through voice mostly uses voice recognition engines for semantic recognition, so as to accurately judge the operator's command intentions. The relevant voice recognition engine is also very mature, and the fastest one can even accurately recognize the operator's command within 0.7 seconds. However, for games, the response speed of 0.7 seconds is still insufficient, especially for action games. How to further speed up the speech recognition speed and realize fast human-computer interaction is a technical problem that needs to be solved urgently. On the other hand, it is difficult for existing smart cockpits to realize simultaneous operation of multiple people, and corresponding multiplayer games are also difficult to apply in smart cockpits. Summary of the invention

[0003] The object of the present invention is to provide a high-speed, efficient, multi-point and multi-person controllable intelligent cockpit human-computer interaction method based on sound source localization.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] The method for human-computer interaction in an intelligent cockpit based on sound source localization includes an intelligent cockpit, wherein the intelligent cockpit is provided with a plurality of sound capture devices for collecting sound and at least one host; the method also includes a plurality of commands for human-computer interaction operations and a pre-set command list, wherein the command list includes all the commands and the sound source position corresponding to each command; the method also includes a sound localization module running in the host for processing the sound and obtaining the sound source position, and a control module for comparing the obtained sound source position with the command list to obtain the corresponding command; the method includes the following steps:

[0006] S100, acquiring sounds in a working environment through a sound capture device;

[0007] S101, the sound localization module processes the obtained sound to obtain the sound source position;

[0008] S102, the control module retrieves the corresponding command from the command list according to the sound source position;

[0009] S103: Perform interactive operations with the host according to the command.

[0010] Preferably, the command list also includes a command phoneme sequence corresponding to each of the commands, and also includes a command recognition method for performing phoneme comparison on the sound collected by the sound capture device to obtain the corresponding command;

[0011] The command recognition method includes the following steps:

[0012] 3.1) Acquire the sound in the working environment through a sound capture device;

[0013] 3.2) The sound obtained in step 3.1) is obtained by the sound processing engine for processing, and background noise is removed therefrom to obtain the control sound therefrom;

[0014] 3.3) recognizing the control sound obtained in step 3.2) by the sound recognition engine, converting the control sound into phonemes, and sequentially composing the phoneme sequence to be recognized;

[0015] 3.4) comparing the to-be-recognized phoneme sequence with the instruction phoneme sequence in the command list through the control module to obtain the corresponding command or comparison failure flag; which includes the following steps:

[0016] 3.4.1) extracting a command from the command list and using the command phoneme sequence corresponding to the command as the command phoneme sequence to be compared;

[0017] 3.4.2) By comparing the phonemes in the to-be-recognized phoneme sequence with the phonemes in the to-be-compared instruction phoneme sequence one by one in order; if the comparison results are inconsistent, execute 3.4.3); if the comparison results are consistent, execute 3.4.4);

[0018] 3.4.3) taking out the next command in the command list, and taking its corresponding command phoneme sequence as the command phoneme sequence to be compared, and executing 3.4.2); if all the commands in the command list have been compared, outputting a recognition failure flag;

[0019] 3.4.4) The counter counts and compares the next phoneme until the number of consistent phonemes exceeds a preset threshold, then the command to be compared is output as the comparison result, and the counter is cleared; if any of the phoneme comparison results are inconsistent, the counter is cleared and 3.4.3) is executed.

[0020] Preferably, the method for presetting the command list comprises the following steps:

[0021] 1.1) Record the sound corresponding to the command;

[0022] 1.2) The sound recognition engine converts the sound corresponding to the command into a number of phonemes;

[0023] 1.3) Combining all or part of the above phonemes in order to obtain an instruction phoneme sequence;

[0024] 1.4) The commands and their corresponding instruction phoneme sequences are stored as the command list.

[0025] Preferably, it also includes the following steps:

[0026] 2.1) In a training environment, a control sound is simulated and acquired through a sound acquisition device, and then the sound processing engine is trained, wherein the training includes removing background noise from the sound acquired by the sound acquisition device and extracting the control sound therefrom; the control sound is composed of a number of words or characters or sounds;

[0027] 2.2) In a training environment, the sound recognition engine is trained, wherein the training content includes converting the control sound obtained by the sound processing engine into phonemes, and sequentially composing the phoneme sequence to be recognized; the phoneme sequence to be recognized is a phoneme sequence composed of a plurality of phonemes in sequence.

[0028] Preferably, the step 2.1) comprises the following steps:

[0029] 2.1.1) Denoising: simulating various types of background noise in a training environment and training the sound processing engine to remove the background noise.

[0030] Preferably, the step 3.2) comprises the following steps:

[0031] 3.2.1) Denoising, removing background noise in the working environment through the sound processing engine, and using the obtained sound as the control sound;

[0032] 3.2.2) If the control sound is a continuous sound, it is continuously transmitted to the sound recognition engine; otherwise, it is completely transmitted to the sound recognition engine;

[0033] 3.2.3) If the control sound is a continuous sound, the sound recognition engine will continuously send the input control sound to the control module, and the control module will execute steps 3.3) and 3.4) in sequence to perform command recognition; wherein, if the control module returns a recognition failure flag, it will be returned to the sound processing engine, and the sound processing engine will stop sending subsequent control sounds to the sound recognition engine until the control sound pauses and a new control sound appears.

[0034] Preferably, it further comprises a sound output device, and the background noise in the step 3.2.1) comprises the sound output by the sound output device.

[0035] Preferably, the threshold is 1-5.

[0036] Preferably, the sound includes the sound produced by clapping, slapping body parts or knocking on the smart cockpit, and the sound produced by operating a sound-generating device.

[0037] The beneficial effects of the present invention are as follows: different operation commands are corresponded by the position of the sound source to realize multi-person or multi-point human-computer interaction; the voice can also be converted into phonemes, and command recognition can be directly performed through some phonemes, thereby reducing the semantic recognition process, which not only simplifies the complexity of the recognition engine, but also greatly improves the recognition speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The schematic diagram is a principle diagram of an application example of the present invention.

[0039] Figure 2 A schematic diagram of a process of an application example of the present invention;

[0040] Figure 3 The figure is a flow chart of an application example of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of this patent is further described in detail below in conjunction with specific implementation methods.

[0042] In the description of the present invention, it should be noted that the terms "inside", "outside", "up", "down", "horizontal", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0043] like Figure 1As shown, the human-computer interaction method of the intelligent cockpit based on sound source localization of the present invention comprises an intelligent cockpit, in which a plurality of sound capture devices for collecting sound and at least one host are arranged, and a display and a speaker may also be configured; it also comprises a plurality of commands for human-computer interaction operations and a pre-set command list, wherein the command list comprises all commands and the sound source position corresponding to each command; it also comprises a sound localization module running in the host for processing the sound and obtaining the sound source position, and a control module for comparing the obtained sound source position with the command list to obtain the corresponding command; it comprises the following steps:

[0044] S100, acquiring sounds in a working environment through a sound capture device;

[0045] S101, the sound localization module processes the obtained sound to obtain the sound source position;

[0046] S102, the control module retrieves the corresponding command from the command list according to the sound source position;

[0047] S103: Perform interactive operations with the host according to the command.

[0048] Its working principle is to use multiple sound capture devices, such as microphones, to form a three-dimensional array to collect sound and determine the location of the sound source. The relevant intelligent microphone array that can directly determine the location of the sound source has mature technology and can be used reasonably as needed. The present invention can also calculate the strength and other characteristics of the same sound collected by different microphones through a reasonable algorithm to infer the location of the sound source. The relevant algorithm is also a mature technology in the industry, so the present invention is not disclosed in detail. The control module of the present invention retrieves the corresponding command in the command list according to the location of the sound source, thereby controlling the host to operate. For example, in a jumping game, the operator can control the character in the game to jump left or right by clapping the left and right legs. Of course, it is also possible to realize a game in which multiple people control multiple characters according to different positions. Furthermore, the sound collected by the sound capture device is also sent to the control module, and the control module can also perform corresponding command operations according to the location, volume, duration, and sound type of the input sound, such as converting the applause from the co-pilot into drum sound, and outputting it synchronously through the speaker; converting the leg clapping sound from the main driver into cymbal sound, and outputting it synchronously through the speaker, thereby realizing instrument ensemble.

[0049] Furthermore, the command list also includes a command phoneme sequence corresponding to each of the commands, and also includes a command recognition method for performing phoneme comparison on the sound collected by the sound capture device to obtain the corresponding command. The intelligent cockpit human-computer interaction method based on sound source localization of the present invention includes a number of commands for human-computer interaction operations, and also includes a pre-set command list, the command list includes all commands and the command phoneme sequence corresponding to each command; it also includes a sound capture device for collecting sound, a sound processing engine for processing sound, a sound recognition engine for recognizing the processed sound, and a control module for comparing the recognized sound with the command list. The above mode further provides convenience for multi-person operation of the game. Operators at different positions in the bus control different characters through voice, thereby perfectly realizing multi-person control and improving entertainment.

[0050] The command recognition method includes a pre-preparation step (which is similar to the work before leaving the factory) and an actual work step. The pre-preparation step includes:

[0051] To pre-set a command list:

[0052] 1.1) Record the sound corresponding to the command;

[0053] 1.2) The voice recognition engine converts the sound corresponding to the command into a number of phonemes;

[0054] 1.3) Combining all or part of the above phonemes in order to obtain an instruction phoneme sequence;

[0055] 1.4) The commands and their corresponding instruction phoneme sequences are stored as a command list.

[0056] It also includes steps to train the engine to improve recognition efficiency and accuracy:

[0057] 2.1) In a training environment, a control sound is simulated and acquired through a sound capture device, and then the sound processing engine is trained, wherein the training includes removing background noise from the sound acquired by the sound capture device and extracting the control sound therefrom; the control sound is composed of a number of words or characters or sounds; and the training includes the steps of:

[0058] 2.1.1) Denoising: simulate various types of background noise in the training environment and train the sound processing engine to remove background noise.

[0059] 2.2) In a training environment, the sound recognition engine is trained, wherein the training content includes converting the sound obtained by the sound processing engine into phonemes, and sequentially composing the phoneme sequences to be recognized; the phoneme sequence to be recognized is a phoneme sequence composed of several phonemes in sequence.

[0060] The actual working steps include:

[0061] 3.1) Acquire the sound in the working environment through a sound capture device;

[0062] 3.2) The sound obtained in step 3.1) is processed by a sound processing engine to remove background noise and obtain a control sound;

[0063] 3.3) Recognizing the control sound obtained in step 3.2) through a sound recognition engine, converting the control sound into phonemes, and sequentially composing the phoneme sequences to be recognized;

[0064] 3.4) The control module compares the to-be-recognized phoneme sequence with the instruction phoneme sequence in the command list to obtain the corresponding command or comparison failure flag; which includes the following steps:

[0065] 3.4.1) extracting a command from the command list and using the command phoneme sequence corresponding to the command as the command phoneme sequence to be compared;

[0066] 3.4.2) By comparing the phonemes in the to-be-recognized phoneme sequence with the phonemes in the to-be-compared instruction phoneme sequence one by one in order; if the comparison results are inconsistent, execute 3.4.3); if the comparison results are consistent, execute 3.4.4);

[0067] 3.4.3) Take out the next command in the command list, and use its corresponding command phoneme sequence as the command phoneme sequence to be compared, and execute 3.4.2); if all commands in the command list have been compared, output a recognition failure flag;

[0068] 3.4.4) The counter counts and compares the next phoneme until the number of consistent phonemes exceeds a preset threshold, then the command to be compared is output as the comparison result, and the counter is cleared; if any of the phoneme comparison results are inconsistent, the counter is cleared and 3.4.3) is executed; wherein the threshold is usually 1 to 5, which is set according to different commands, so that the corresponding command can be found without a complete comparison of the phoneme list.

[0069] in:

[0070] Step 3.2) includes the following steps:

[0071] 3.2.1) Denoising: removing background noise in the working environment through the sound processing engine, and using the obtained sound as the control sound;

[0072] 3.2.2) If the control sound is a continuous sound, it is continuously transmitted to the sound recognition engine; otherwise, it is completely transmitted to the sound recognition engine;

[0073] 3.2.3) If the control sound is a continuous sound, the sound recognition engine will continuously send the input control sound to the control module, and the control module will execute steps 3.3) and 3.4) in sequence to perform command recognition; wherein, if the control module returns a recognition failure flag, it will be returned to the sound processing engine, and the sound processing engine will stop sending subsequent control sounds to the sound recognition engine until the control sound pauses and then a new control sound appears. For example, if someone in the cockpit is saying non-control and continuous words, the sound recognition engine can know from the first few words that it is a non-control command, and the subsequent continuous sound will be discarded by the sound processing engine to save resources. If a pause occurs in the middle, it will return to the initial state and wait for sound input for recognition.

[0074] Furthermore, it also includes a sound output device, and the background noise in step 3.2.1) includes the sound output by the sound output device, which makes it possible to exclude the sound output by the game itself through the speaker as background noise.

[0075] The present invention also provides a gaming cockpit, which includes at least one seat, a sound output device, a display, at least one sound capture device, and a host for running a game. The host uses an intelligent cockpit human-computer interaction method based on sound source positioning to convert the voice of the game player into a command for controlling the game, and executes the game operation according to the command.

[0076] The sounds made by the game player include clapping, slapping body parts or knocking on bus parts, and operating a sound-generating device (such as a bell).

[0077] like Figure 1 As shown, taking the common game command stop as an example, the working principle and process of the present invention are as follows:

[0078] First, the sound of the user's stop is acquired by a sound capture device (such as a microphone), and then the sound recognition engine converts it into a phoneme list of / s, / t, / oh, and / p. The technology of converting sound into phonemes by the sound recognition engine is well known in the industry, and the relevant algorithm is very mature. Please refer to patent 201711082646.9. In the pre-set command list, there is a corresponding phoneme sequence. Through a simple comparison, the corresponding instruction stop can be found. In the game, the number of operation instructions is relatively small, so it is possible to obtain a unique operation instruction stop after comparing / s or / s, / t, so as to control the game and achieve the purpose of rapid recognition. The threshold is determined according to the different operation instructions. If 3 phonemes can uniquely determine different operation instructions, the threshold is set to 3, which can reduce the number of phoneme comparisons and avoid unnecessary redundant comparisons to further improve the speed.

[0079] like Figure 2 As shown, taking the common game commands left, right, stop, and start as examples, the working principle and process of the present invention are as follows: the sound recognition engine first converts the sound into a phoneme sequence / left, / right, / stop, / start; then through a simple screening, for example, after comparing / s, it further compares / t; if it still cannot obtain a unique command, it further compares / o or / a to obtain a unique command. For the above commands, the threshold can be set to 3 at this time, and a unique command can be obtained by comparing 3 phonemes. Similarly, only one phoneme is actually needed for comparing / left and / right, namely / l and / r. When the comparison reaches the threshold of 3 at most, the result can be returned, and the comparison of other commands is abandoned, thereby increasing the comparison speed.

[0080] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A human-computer interaction method for an intelligent cockpit based on sound source localization, comprising an intelligent cockpit, characterized in that: The smart cockpit is provided with a plurality of sound capture devices for collecting sound and at least one host; it also includes a plurality of commands for human-computer interaction and a pre-set command list, wherein the command list includes all the commands and the sound source position corresponding to each command; it also includes a sound positioning module running in the host to process the sound and obtain the sound source position, and a control module to compare the obtained sound source position with the command list to obtain the corresponding command; It includes the following steps: S100, acquiring sounds in a working environment through a sound capture device; S101, the sound localization module processes the obtained sound to obtain the sound source position; S102, the control module retrieves the corresponding command from the command list according to the sound source position; S103, performing interactive operations with the host according to the command; The command list also includes a command phoneme sequence corresponding to each of the commands, and also includes a command recognition method for performing phoneme comparison on the sound collected by the sound capture device to obtain the corresponding command; The command recognition method includes the following steps: 3.1) Acquire the sounds in the working environment through a sound capture device; 3.2) The sound obtained in step 3.1) is processed by a sound processing engine to remove background noise and obtain the control sound; 3.3) Recognizing the control sound obtained in step 3.2) through a sound recognition engine, converting the control sound into phonemes, and sequentially composing the phoneme sequences to be recognized; 3.4) The control module compares the to-be-recognized phoneme sequence with the instruction phoneme sequence in the command list to obtain the corresponding command or comparison failure flag; which includes the following steps: 3.4.1) extracting a command from the command list and using the command phoneme sequence corresponding to the command as the command phoneme sequence to be compared; 3.4.2) By comparing the phonemes in the to-be-recognized phoneme sequence with the phonemes in the to-be-compared instruction phoneme sequence one by one in order; if the comparison results are inconsistent, execute 3.4.3); if the comparison results are consistent, execute 3.4.4); 3.4.3) taking out the next command in the command list, and using its corresponding command phoneme sequence as the command phoneme sequence to be compared, and executing 3.4.2); if all the commands in the command list have been compared, outputting a recognition failure flag; 3.4.4) The counter counts and compares the next phoneme until the number of consistent phonemes exceeds a preset threshold, then the command to be compared is output as the comparison result, and the counter is cleared. The threshold setting must be sufficient to uniquely identify different operation instructions; if any of the phoneme comparison results are inconsistent, the counter is cleared and 3.4.3) is executed.

2. The intelligent cockpit human-computer interaction method based on sound source localization according to claim 1 is characterized in that: The method for presetting the command list comprises the following steps: 1.1) Record the sound corresponding to the command; 1.2) The sound recognition engine converts the sound corresponding to the command into a number of phonemes; 1.3) Combining all or part of the above phonemes in order to obtain an instruction phoneme sequence; 1.4) The commands and their corresponding instruction phoneme sequences are stored as the command list.

3. The intelligent cockpit human-computer interaction method based on sound source localization according to claim 2 is characterized in that: It also includes the following steps: 2.1) In a training environment, a control sound is simulated and acquired through a sound acquisition device, and then the sound processing engine is trained, wherein the training includes removing background noise from the sound acquired by the sound acquisition device and extracting the control sound therefrom; the control sound is composed of a plurality of words or characters; 2.2) In a training environment, the sound recognition engine is trained, wherein the training includes converting the control sound obtained by the sound processing engine into phonemes, and sequentially composing the phoneme sequences to be recognized; the phoneme sequence to be recognized is a phoneme sequence composed of a plurality of phonemes in sequence.

4. The intelligent cockpit human-computer interaction method based on sound source localization according to claim 3 is characterized in that: Step 2.1) The following steps are involved: 2.1.1) Denoising: simulating various types of background noise in a training environment and training the sound processing engine to remove the background noise.

5. The intelligent cockpit human-computer interaction method based on sound source localization according to claim 2 is characterized in that: The step 3.2) comprises the following steps: 3.2.1) Denoising, removing background noise in the working environment through the sound processing engine, and using the obtained sound as the control sound; 3.2.2) If the control sound is a continuous sound, it is continuously transmitted to the sound recognition engine; otherwise, it is completely transmitted to the sound recognition engine; 3.2.3) If the control sound is a continuous sound, the sound recognition engine continuously sends the input control sound to the control module, and the control module sequentially executes steps 3.3) and 3.4) to perform command recognition; wherein, if the control module returns a recognition failure flag, it returns it to the sound processing engine, and the sound processing engine stops sending subsequent control sounds to the sound recognition engine until the control sound pauses and then a new control sound appears.

6. The intelligent cockpit human-computer interaction method based on sound source localization according to claim 5 is characterized in that: It also includes a sound output device, and the background noise in step 3.2.1) includes the sound output by the sound output device.

7. The intelligent cockpit human-computer interaction method based on sound source localization according to claim 1 is characterized in that: The threshold value is 1-5.

8. The intelligent cockpit human-computer interaction method based on sound source localization according to claim 1, characterized in that: The sounds include clapping, slapping of body parts or tapping of the smart cockpit, and sounds produced by operating a sound-generating device.

Images