Voice interaction methods, electronic devices and media

Abstract

This application relates to the field of voice control, and discloses a voice interaction method, electronic device, and medium. The voice interaction method includes: when it is detected that the current voice interaction environment of the electronic device does not meet the conditions for voice recognition, determining whether the user's current interaction state meets the conditions for lip reading; if it is determined that the conditions are met, acquiring the lip reading result obtained by recognizing the user's voice command received by the electronic device using lip reading; and executing the function corresponding to the lip reading result. The voice recognition method provided in this application, when it is determined that voice recognition alone is insufficient to recognize the user's voice command, further determines whether to use lip reading by judging whether the user is interacting with a voice assistant; this method can effectively improve the accuracy of voice command recognition, thereby further improving the accuracy of the electronic device in executing the user's voice command.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a voice interaction method, electronic device and medium. Background Technology

[0002] With the development of artificial intelligence technology, robots and other electronic devices have been widely used in industries such as catering, education, healthcare, culture, smart homes, finance, and telecommunications, providing users with comprehensive artificial intelligence services.

[0003] Users can interact with robots and other electronic devices through touchscreens, voice commands, remote controls, and other methods. When users interact with robots using voice, the robots can recognize and execute the user's voice commands. For example... Figure 1 As shown, if a user wants to command the robot to perform the "storytelling" operation, they can issue the voice command "storytelling". After recognizing the voice command "storytelling", the robot can perform the "storytelling" operation.

[0004] However, in noisy environments, when a user gives commands to the robot via voice, the robot may be unable to receive or recognize the user's voice commands and execute them. For example, if the environment is too noisy, the robot may not be able to determine when the user's voice command ends, thus remaining in a continuous recording state; or it may not be able to determine when the user's voice command begins, thus remaining in a non-recording state. Therefore, it cannot provide feedback such as executing user voice commands, which seriously affects the user experience. Summary of the Invention

[0005] To address the technical problem that electronic devices may be unable to receive or recognize and execute user voice commands in noisy environments, this application provides a voice interaction method, an electronic device, and a medium in its first embodiment. This method can be applied to electronic devices and includes:

[0006] If the current voice interaction environment of the electronic device does not meet the voice recognition conditions, determine whether the user's current interaction state meets the lip reading recognition conditions.

[0007] If the user's current interaction state is determined to meet the lip reading recognition conditions, the lip reading recognition result is obtained by recognizing the user's mouth change features acquired by the electronic device through the image acquisition device using the lip reading recognition method.

[0008] Perform the function corresponding to the lip reading result.

[0009] The speech recognition method provided in this application, when it is determined that speech recognition alone is insufficient to recognize the user's voice commands, further determines whether to use lip reading recognition by judging whether the user is interacting with a voice assistant. This method can effectively improve the accuracy of speech command recognition, thereby further improving the correctness of electronic devices in executing user voice commands.

[0010] It is understood that, in the embodiments of this application, the noise level around the electronic device and the duration of the electronic device's audio reception are both within the scope of the electronic device's voice interaction environment.

[0011] It is understood that in the embodiments of this application, the image acquisition device can be a camera device used to acquire images, such as a webcam.

[0012] It is understood that in the embodiments of this application, when the electronic device receives the user's voice command, it can simultaneously collect the user's voice and the user's mouth change features.

[0013] When it is determined that the current voice interaction environment of the electronic device meets the conditions for voice recognition, the voice recognition method is used to identify the user's voice received by the electronic device to obtain the voice recognition result.

[0014] When it is determined that the current voice interaction environment of the electronic device does not meet the conditions for voice recognition, but the current interaction state of the user meets the conditions for lip reading, lip reading can be used to identify the changes in the user's mouth acquired by the image acquisition device of the electronic device to obtain the lip reading results.

[0015] In one possible implementation of the first aspect described above, the speech recognition conditions include:

[0016] The noise level around the electronic device is lower than the set noise level.

[0017] or;

[0018] When the noise level around the electronic device is greater than or equal to a set noise level, the duration of the electronic device's sound reception is greater than zero and less than the set time.

[0019] It is understood that in some embodiments, determining whether a user's voice command can be recognized by voice recognition can first determine whether the noise level around the electronic device is lower than a set noise level. If so, it proves that the noise around the electronic device is relatively low, and the user's voice command can be recognized by voice recognition. If not, it proves that the noise around the electronic device is relatively high and the external environment is noisy. In this case, it is further determined that the continuous sound received by the electronic device is greater than zero and less than a set time.

[0020] If the continuous sound from the electronic device is greater than zero and less than the set time, it proves that the electronic device can still accurately determine the time point when the user's voice is interrupted, and thus determine that the voice recognition method can be used to recognize the user's voice command at this time. If the continuous sound from the electronic device is greater than or equal to the first set value or less than zero, it means that the electronic device can no longer accurately determine the time point when the user's voice is interrupted, and thus determine that the voice recognition method can hardly recognize the user's voice command at this time.

[0021] In one possible implementation of the first aspect above, the speech recognition condition includes: the duration of the electronic device's sound reception is greater than zero and less than a set time.

[0022] It is understood that in some embodiments, determining whether voice recognition can be used to recognize a user's voice command can be achieved by directly judging whether the duration of the electronic device's audio reception is greater than zero and less than a set time. If so, it proves that the electronic device can still accurately determine the time point when the user's voice is interrupted, and thus determine that voice recognition can be used to recognize the user's voice command at this time. If the duration of the electronic device's audio reception is greater than or equal to a first set value or less than zero, it indicates that the electronic device can no longer accurately determine the time point when the user's voice is interrupted, thus inferring that the external environment is too noisy, and determining that voice recognition is no longer sufficient to recognize the user's voice command at this time.

[0023] In one possible implementation of the first aspect described above, the lip-reading conditions include:

[0024] Users are in an interactive state with electronic devices for a set period of time.

[0025] It is understandable that when a user is constantly interacting with an electronic device, it means that the user still needs the electronic device to execute voice commands. If the user is no longer interacting with the electronic device, for example, if they have left, it means that the user no longer needs the electronic device to execute voice commands.

[0026] In some embodiments, if the user is constantly interacting with the electronic device, the electronic device can capture the user's mouth changes more clearly, making it easier to use lip reading to identify the acquired mouth changes and obtain lip reading results.

[0027] In one possible implementation of the first aspect described above, the method for determining whether the user and the electronic device are in an interactive state within a set time period includes:

[0028] Determine whether the user interacting with the electronic device within the set time period has changed;

[0029] Detect whether the interaction strength value between the user and the electronic device reaches a set strength value;

[0030] If it is determined that the user interacting with the electronic device within the set time period has not changed, and the interaction intensity value between the user and the electronic device reaches the set intensity value, then it is confirmed that the user and the electronic device are in an interactive state.

[0031] The interaction intensity value is related to the distance between the user and the electronic device and the orientation of the user's face.

[0032] In some embodiments of this application, before detecting the interaction intensity value between the user and the electronic device, it can be determined whether the user interacting with the electronic device within the aforementioned set time period has changed. If the user has not changed, the current user can be used as the object for subsequent detection of the interaction intensity value. When it is further determined that the interaction intensity value between the user and the electronic device reaches the set intensity value, it can be confirmed that the user and the electronic device are in an interactive state; it can be determined that at this time, using lip reading can more accurately identify the user's mouth changes to obtain lip reading results.

[0033] It is understood that in this embodiment of the application, the interaction intensity value can be obtained based on the distance between the user's face and the electronic device, the orientation of the face, etc., within a set time period. For example, if the distance between the user's face and the electronic device is relatively close within a set time period, and the user's face is facing the electronic device, the interaction intensity value is higher; otherwise, it is weaker.

[0034] It is understood that the interaction intensity value and interaction willingness value mentioned in the embodiments of this application have the same meaning, only expressed differently. Furthermore, the intensity value can be the second set value mentioned in the embodiments below.

[0035] In one possible implementation of the first aspect described above, before executing the functional step corresponding to the lip-reading result, the method further includes:

[0036] Confirm that the lip reading results are correct;

[0037] If the lip reading result is confirmed to be correct, the function corresponding to the lip reading result is executed.

[0038] In this embodiment of the application, before executing the functional steps corresponding to the lip reading result, the lip reading result is confirmed again to ensure its correctness, which can effectively improve the accuracy of the electronic device in executing the user's voice commands.

[0039] In one possible implementation of the first aspect described above, the method for confirming whether the lip reading result is correct includes:

[0040] Ask the user whether they need to execute the function corresponding to the lip reading result;

[0041] If the user confirms that the function corresponding to the lip reading result needs to be executed, the lip reading result is confirmed to be correct.

[0042] In one possible implementation of the first aspect described above, the step of confirming whether the lip-reading result is correct also includes:

[0043] The user's body movement characteristics and the noise level around the electronic device are obtained.

[0044] In this embodiment of the application, in the step of confirming whether the lip reading result is correct, since the aforementioned steps have confirmed that the current environment is noisy, the voice recognition method may no longer be able to accurately recognize the user's voice commands. Therefore, when the voice assistant confirms with the user, the visual recognition function can be enabled at the same time. The visual recognition function can obtain the user's body movement characteristics and facilitate the recognition of the user's response through body movements. For example, the user may indicate that the lip reading result is correct by nodding or making an OK gesture.

[0045] In some embodiments, to further enhance the accuracy of the electronic device in recognizing user voice commands, in addition to enabling visual recognition, a noise detection function can also be activated when making voice confirmations to the user, in order to detect ambient noise in real time. If the ambient noise is determined to be below a set value, it can be determined that the voice recognition method can recognize the user's voice command, and then voice recognition can be used to recognize the user's confirmation command or subsequent voice commands. If the ambient noise is still above the set value, lip reading, visual recognition, or a combination of lip reading and visual recognition can be used to recognize the user's confirmation command or subsequent voice commands.

[0046] In one possible implementation of the first aspect described above, the electronic device is a robot.

[0047] The speech recognition method provided in the first aspect of this application, when it is determined that speech recognition alone is insufficient to recognize the user's voice commands, further determines whether to use lip reading recognition by judging whether the user is interacting with a voice assistant. This method can effectively improve the accuracy of speech command recognition, thereby further improving the accuracy of electronic devices in executing user voice commands.

[0048] A second aspect of this application provides an electronic device, including:

[0049] A memory for storing instructions executed by one or more processors of the electronic device, and

[0050] The processor is one of the one or more processors in the electronic device, used to execute the above-described voice interaction method.

[0051] A third aspect of this application provides a computer-readable storage medium storing instructions that, when executed, cause a computer to perform the aforementioned voice interaction method.

[0052] A fourth aspect of this application provides a computer program product, the computer program product including instructions that, when executed, cause a computer to perform the above-described voice interaction method. Attached Figure Description

[0053] Figure 1 According to some embodiments of this application, a scenario diagram of a voice interaction method is shown;

[0054] Figure 2 According to some embodiments of this application, a schematic diagram of the structure of an electronic device is shown;

[0055] Figure 3 According to some embodiments of this application, a flowchart of a voice interaction method is shown;

[0056] Figure 4 According to some embodiments of this application, a scenario diagram of a voice interaction method is shown;

[0057] Figure 5 According to some embodiments of this application, a scenario diagram of a voice interaction method is shown;

[0058] Figure 6 According to some embodiments of this application, a flowchart of a voice interaction method is shown. Detailed Implementation

[0059] Embodiments of this application disclose a voice interaction method, an electronic device, and a medium.

[0060] It is understood that the electronic devices applicable to the embodiments of this application can be various electronic devices with voice recognition capabilities, including but not limited to robots, laptop computers, desktop computers, tablet computers, smartphones, servers, wearable devices, head-mounted displays, mobile email devices, portable game consoles, portable music players, e-reader devices, televisions in which one or more processors are embedded or coupled, or other electronic devices with computing capabilities.

[0061] Furthermore, the voice recognition function of the aforementioned electronic devices can be implemented in the form of various applications, such as a voice assistant, or the voice recognition function can be built into the applications of the electronic devices, such as voice search for applications, such as voice search in map applications.

[0062] For ease of explanation, the following text uses electronic devices as robots, and the voice recognition function is implemented using the robot's voice assistant as an example.

[0063] As mentioned earlier, users can control electronic devices such as robots using voice commands. However, in noisy environments, if the robot's voice assistant uses voice recognition to recognize the user's voice commands, it may fail to determine when the user's voice command ends, thus remaining in a recording state, or it may fail to determine when the user's voice command begins, thus remaining in a non-recording state. This can result in the inability to execute the user's voice commands, thereby affecting the user experience.

[0064] To address the aforementioned issues, this application provides a voice interaction method. When a voice assistant is activated by a user, it can detect the ambient noise level using a noise detection function. If the noise level exceeds a set threshold, the current voice recognition method is switched to lip reading, enabling the electronic device to recognize and execute the user's voice commands through lip reading technology.

[0065] For example, with Figure 1 Taking the scenario shown as an example, when user 001 issues the voice command "tell a story", during the process of the user issuing the command, the voice assistant detects that the ambient noise level is high, exceeding the set threshold, for example, the set threshold is 70 decibels. If the voice assistant detects that the ambient noise level is 78 decibels, the voice assistant will switch the current voice recognition method to lip reading, so that the voice assistant can recognize the user's voice command through lip reading technology and execute the "tell a story" voice command.

[0066] The aforementioned technologies can recognize voice commands under certain conditions. However, in normal scenarios, the accuracy of lip reading is generally lower than that of voice recognition. Therefore, the above solutions convert voice recognition to lip reading when there is high noise. There may be scenarios where voice recognition can still be accurate even when the surrounding environment is noisy. In such cases, converting voice recognition to lip reading increases the risk of recognition errors.

[0067] Therefore, this application provides another voice interaction method. This method does not directly switch the voice recognition mode to lip reading mode and obtain lip reading results after determining that the surrounding environment is too noisy through noise detection; instead, it first determines whether voice recognition cannot be used, and if so, whether the conditions for using lip reading are met. Only after determining that the conditions for lip reading are met is the lip reading result obtained. The determination of whether voice recognition cannot be used is as follows:

[0068] In one feasible solution, it can be determined that the voice assistant's recording time is too long, for example, exceeding the system's normal setting value. This indicates that the external environment may be too noisy, preventing the voice assistant from determining the end time of the user's voice command. Alternatively, it can be determined that the inability to record audio indicates that the external environment may be too noisy, preventing the voice assistant from determining the start time of the user's voice command. In such cases, it can be concluded that voice recognition is insufficient to accurately identify the user's voice command, and therefore, voice recognition should not be used to perform voice recognition on the user.

[0069] In another feasible solution, the surrounding environment can be assessed first using a noise detection function. If the noise level is lower than a set noise level, it can be determined that voice recognition can be used. If the noise level is greater than or equal to the set noise level, it can be further determined whether the voice assistant has been recording for too long, such as exceeding the system's normal setting, or whether it is unable to record. This indicates that the external environment is so noisy that the voice assistant cannot determine the start or end time of the user's voice command, thus making it difficult to accurately recognize the user's voice command through voice recognition.

[0070] After determining that speech recognition is insufficient to identify the user's voice commands, the system determines whether to use lip reading by checking if the user's face is facing the camera or within the camera's field of view within a set time period. If the user is confirmed to be interacting with the voice assistant, lip reading can be used to relatively accurately identify the user's voice commands. The lip reading result is then obtained, and the corresponding function is executed based on the lip reading result.

[0071] For example, with Figure 1Taking the scenario shown as an example, when the voice assistant is activated, it enters voice recognition mode and begins recording. When user 001 issues the voice command "tell a story," the voice assistant does not detect that the user's voice command has ended and continues recording. When the recording time exceeds the system's set value, such as 10 seconds, the voice assistant can determine that the external environment may be too noisy, preventing it from determining the end time of user 001's voice command. Therefore, it is determined that voice recognition alone is insufficient to accurately identify the user's voice command. Then, it checks whether user 001's face has been facing the camera and within the robot's camera's field of view throughout the recording process. If the result is yes, it is determined that user 001 is interacting with the electronic device. This allows it to determine whether lip reading can accurately recognize user 001's voice command. Therefore, lip reading can be used to recognize user 001's voice command and execute the "tell a story" voice command.

[0072] The speech recognition method provided in this application first determines that the voice assistant has been receiving sound for a considerable period of time, indicating that the voice assistant may no longer be able to determine when the user's voice command has ended, thus indicating that the external environment is too noisy. Furthermore, it more accurately determines that speech recognition is insufficient to recognize the user's voice command in this situation, and further determines whether to use lip reading by judging whether the user is interacting with the voice assistant. This effectively avoids the reduced accuracy caused by using lip reading when speech recognition is possible, thus effectively improving the accuracy of speech command recognition.

[0073] Before detailing another voice interaction method provided in the embodiments of this application, the electronic device provided in the embodiments of this application will be introduced first.

[0074] For ease of explanation, we will now use robot 002 as an example. It should be understood that robot 002 in this embodiment can also interact with a cloud server, sending the cloud commands identified by user 001 to the cloud server. The cloud server can use a database to provide feedback on the interaction content to robot 002, such as songs, stories, etc.

[0075] like Figure 2 As shown, robot 002 may include processor 110, power module 140, memory 180, sensor module 190, audio module 150, camera 170, interface module 160, button 101, and display screen 102, etc.

[0076] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on robot 002. In other embodiments of this application, robot 002 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0077] Processor 110 may include one or more processing units, such as processing modules or circuits of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), neural network processing unit (NPU), microprocessor (MCU), AI (Artificial Intelligence) processor, or field programmable gate array (FPGA). Different processing units may be independent devices or integrated into one or more processors. Processor 110 may include storage units for storing instructions and data. In some embodiments, the storage unit in processor 110 is a cache memory 180.

[0078] It is understood that in this embodiment, the processor 110 can control the corresponding program to execute the voice interaction method provided in this embodiment. Specifically, an artificial intelligence processor can be used to recognize the received voice and obtain the recognition result; the aforementioned image processor can be used to analyze the captured lip movements of user 001 and obtain the recognition result; simultaneously, the aforementioned image processor can be used to recognize the captured body movements of user 001 and obtain the recognition result. Additionally, the processor 110 can detect noise around the electronic device in real time to select a more accurate recognition method.

[0079] The power module 140 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component manages the charging of the power supply and the power supply to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module receives charging input from a charger; the power management module connects to the power supply and the processor 110. The power management module receives input from the power supply and / or the charging management module to supply power to the processor 110, the display 102, the camera 170, and the wireless communication module 120, etc.

[0080] The wireless communication module 120 may include an antenna, which enables the transmission and reception of electromagnetic waves. The wireless communication module 120 can provide solutions for wireless communication applications on the robot 002, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The robot 002 can communicate with networks and other devices via wireless communication technology. For example, the robot 002 can communicate with a cloud server through the wireless communication module 120.

[0081] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In this embodiment, the display screen 102 can be used to display various application interfaces of the robot 002.

[0082] The sensor module 190 may include proximity sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0083] The audio module 150 is used to convert digital audio information into analog audio signals for output, or to convert analog audio input into digital audio signals. The audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 150 can be located in the processor 110, or some functional modules of the audio module 150 can be located in the processor 110. In some embodiments, the audio module 150 may include a speaker, a handset, a microphone, and a headphone jack. In this embodiment, the audio module 150 can be used to receive voice commands from user 001, and, if feasible, the audio module 150 can also be used to perform operations such as playing music or telling stories based on the voice commands of user 001.

[0084] Camera 170 is used to capture still images or videos. An object generates an optical image through the lens and projects it onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP (Image Signal Processing) to be converted into a digital image signal. Robot 002 can achieve its shooting function through the ISP, camera 170, video codec, GPU (Graphics Processing Unit), display screen 102, and application processor. In this embodiment, camera 170 can acquire images of user 001's face, lip movements, etc.

[0085] The interface module 160 includes an external memory interface and a universal serial bus (USB) interface. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the robot 002. The external memory card communicates with the processor 110 through the external memory interface to perform data storage. The universal serial bus interface is used for communication between the robot 002 and other electronic devices 002.

[0086] In some embodiments, robot 002 further includes a button 101. The button 101 may include volume buttons, power on / off buttons, etc.

[0087] The following describes in detail another voice interaction method of the present application embodiment with reference to the robot 002 described above. Figure 3 A schematic diagram of a voice interaction method is shown, wherein, Figure 3 The voice interaction method shown can be executed by the voice assistant of robot 002. For example... Figure 3 As shown, Figure 3 The voice interaction methods shown include:

[0088] S301: After detecting that user 001 has woken up the voice assistant, enter the voice recognition mode. In this embodiment, the voice recognition mode is the mode for recognizing the voice commands received by user 001 from robot 002.

[0089] After the voice assistant is activated, it immediately begins recording sound and acquiring the lip movements of user 001. Recording sound facilitates the recognition of user 001's voice commands in subsequent steps, allowing for direct identification of the received audio. Acquiring the lip movements of user 001 facilitates the recognition of user 001's voice commands using lip reading in subsequent steps, enabling direct identification of the acquired lip movements.

[0090] In this embodiment of the application, sound reception can be achieved using the microphone of robot 002, and the acquisition of mouth change features of user 001 can be achieved using the camera device of robot 002.

[0091] It is understood that in some embodiments, the voice assistant of robot 002 needs to receive a wake-up word from user 001 before it can be activated and enter speech recognition mode. For example, if the wake-up word of the voice assistant is "Hi, Xiaoyi", when user 001 says "Hi, Xiaoyi", the voice assistant will enter speech recognition mode, turn on sound recording and obtain the user's mouth change features in order to receive the user 001's voice commands.

[0092] In this embodiment of the application, during the operation of the speech recognition mode, the speech recognition function and the lip reading function can be selectively enabled. As described below:

[0093] In one feasible solution, both speech recognition and lip reading functions can be activated throughout the entire sound recording process. This allows for real-time speech recognition of the received sound and real-time lip reading of the captured user's mouth features. If the subsequent decision indicates the use of speech recognition, no further speech recognition is needed; similarly, if the subsequent decision indicates the use of lip reading, no further lip reading is needed, effectively saving time spent on recognizing the user's voice.

[0094] In the second feasible solution, after entering the speech recognition mode, the system can determine in subsequent steps whether to use lip reading or speech recognition to recognize the user's voice commands before activating the speech recognition or lip reading function. This solution can effectively reduce unnecessary computational load for speech recognition or lip reading.

[0095] In the third feasible solution, speech recognition can be enabled throughout the entire sound recording process, and lip-reading can be enabled only when it is determined that lip-reading is required. This implementation is based on the fact that speech recognition is used to recognize user commands in most scenarios. Therefore, keeping speech recognition enabled for an extended period effectively avoids multiple switching between enabling and disabling it, reducing the computational load on the processor 110 in most common scenarios and improving its operating speed. Lip-reading is only used in a few scenarios; therefore, enabling it only when it is determined to use lip-reading effectively reduces the computational load of lip-reading, thereby reducing the overall computational load of the speech recognition process.

[0096] It is understood that in this embodiment, the speech recognition function can be implemented based on the artificial intelligence processor of robot 002. The artificial intelligence processor can intelligently recognize the speech of user 001. The lip-reading function can be implemented based on the image processor of robot 002. The image processor can continuously identify faces in an image, determine the person speaking, and extract the continuous lip-reading features of that person. These continuously changing features are then input into the lip-reading model in the image processor to identify the pronunciation corresponding to the speaker's lip movements. Subsequently, based on the identified pronunciation, the most probable natural language sentence is obtained.

[0097] It's understandable that lip-reading recognition doesn't have a high accuracy rate in general scenarios. However, in specific scenarios (i.e., recognizing keywords trained by a model), the accuracy is quite high, for example, reaching over 90%.

[0098] S302: Detect whether the conditions for using speech recognition to recognize the user's voice commands are met.

[0099] If the conditions are met, it indicates that the voice recognition method can recognize the user's voice commands, and proceed to S308 to obtain the voice recognition result; if the conditions are not met, it indicates that the voice recognition method can no longer accurately recognize the user's voice commands, and proceed to S303 to check whether the conditions for entering the lip reading recognition mode are met.

[0100] In this embodiment of the application, the voice assistant can control the microphone to pick up sound, and can determine the time point when the user's voice starts or stops by using voice activity detection (Vad) technology.

[0101] In one feasible solution, the condition for recognizing user 001's voice commands using voice recognition is that the voice assistant has received the sound and the sustained sound is less than a first set value.

[0102] The first setting value can be determined based on the relevant performance of the device and the normal voice duration of user 001. For example, although the device can still recognize voices longer than 10 seconds, the device's dialogue system can no longer provide a valid response; or, in other words, the time for user 001 to issue a voice command generally does not exceed 10 seconds, and there will generally be a pause within 10 seconds. When the continuous sound reception exceeds 10 seconds, the device's VAD technology can no longer accurately identify the cutoff point of human voice in the audio. Therefore, the first setting value can be set to 10 seconds.

[0103] It is understood that, in one scenario, when the voice assistant continues to receive audio for a period exceeding a first preset value, it can be determined that the voice boundary detection technology can no longer accurately determine the end point of user 001's voice, hence the voice assistant continues to receive audio. This indicates that the external environment is too noisy, and it can be determined that at this time, it is difficult to recognize user 001's voice commands through voice recognition.

[0104] It is understood that in this embodiment of the application, "enabling sound reception" can mean that the sound reception function is enabled and sound reception is allowed. However, in some cases, there may still be situations where sound reception is not possible. For example, the speech boundary detection technology can no longer accurately determine the start time of the user's 001 voice, so the voice assistant may be unable to receive sound.

[0105] Therefore, in another scenario, when the voice assistant consistently fails to pick up audio, it can be determined that the voice boundary detection technology is no longer able to accurately pinpoint the start time of user 001's voice command, hence the voice assistant's inability to pick up audio. This indicates that the external environment is too noisy, and that speech recognition methods are no longer able to accurately recognize user 001's voice commands.

[0106] The above solution determines that the voice assistant has started receiving sound and the continuous sound reception is less than or equal to a first set value. This confirms that the VAD technology can accurately determine the start and end points of user 001's voice, and thus determines that voice recognition can be used to recognize user 001's voice commands at this time.

[0107] In another feasible approach, the condition for recognizing user 001's voice commands using voice recognition can be that the ambient noise level is less than a set value.

[0108] It should be noted that when the ambient noise level exceeds a set value, it indicates that the external environment is noisy. However, this does not immediately mean that speech recognition cannot be used. Instead, it further determines whether the duration of the voice assistant's audio reception is less than a first set value. As mentioned earlier, this indicates that the VAD technology can still accurately determine the timing of user 001's voice interruption, thus confirming that speech recognition can be used to recognize user 001's voice commands. If the duration of the voice assistant's audio reception is greater than or equal to the first set value, as mentioned above, this indicates that the VAD technology can no longer accurately determine the timing of user 001's voice interruption, thus confirming that speech recognition can no longer accurately recognize user 001's voice commands.

[0109] S303: Detect whether the conditions for using lip reading to recognize the user's voice commands are met.

[0110] If the condition is met, it indicates that lip reading can recognize the user's voice commands, and the process proceeds to S304 to obtain the lip reading result; if the condition is not met, it indicates that lip reading cannot recognize the user's voice commands, and the process proceeds to S307 to prompt the user that the recognition failed.

[0111] It is understood that in some embodiments of this application, the conditions for entering the lip-reading recognition mode are met only when the voice assistant detects that user 001 maintains a continuous interaction with the voice assistant, for example, user 001 in front of robot 002 remains unchanged, and the user's face is always facing the camera device of robot 002, and the distance between robot 002 and robot 002 is within a set range. This is because if user 001 in front of robot 002 is always the same user, and the user's face is always facing the camera device of robot 002, and the distance between the user and robot 002 is within the set range, then the user's mouth movements can be accurately captured. In this case, lip-reading recognition can more accurately identify user 001's voice commands.

[0112] S304: Obtain lip reading results.

[0113] In this embodiment, the lip-reading result can be the instruction keywords included in the user's 001 command. These keywords can be pre-trained, conventional instruction keywords stored in the voice assistant, such as "tell a story," "read a picture book," "play music," "tell a joke," "exit," "return," etc. It is understood that because these keywords are already stored in the voice assistant, lip-reading can accurately identify them.

[0114] In some embodiments, to avoid misidentification in scenarios where command words are mixed in with long sentences, the keyword can be used as the lip reading result only if there is a pause before or after the keyword is identified.

[0115] For example, if user 001 first gives the voice command "tell a story," and then later says, "I heard that robot 002 also has functions like playing music," user 001 actually wants robot 002 to tell a story, not play music. If the voice assistant treats both "tell a story" and "play music" as lip-reading results, it will be unable to determine which lip-reading result corresponds to which function, or it might directly execute the function corresponding to the second result, "play music." However, if the voice assistant can only treat a keyword as a lip-reading result if there is a pause before or after it, then it can directly use "tell a story" as the lip-reading result, thus executing the function corresponding to that result. This solution effectively avoids misrecognition that might occur when command words appear in long sentences.

[0116] S305: Confirm whether the lip reading result is correct. If the result is yes, it indicates that the lip reading is accurate, and proceed to S305; if the result is no, it indicates that the lip reading is incorrect, and proceed to S306.

[0117] To further confirm the accuracy of the lip-reading results, the speech recognition method provided in this application embodiment may include a step of confirming a voice command with user 001. Confirming the voice command with user 001 may involve asking user 001 whether they want the voice assistant to perform the function corresponding to the lip-reading result. For example, as... Figure 4 As shown, if the identified keyword is "storytelling", the way to confirm the voice command with user 001 can be to ask user 001, and the content of the question can be: "Do you want me to tell you a story?" etc.

[0118] When user 001 confirms a voice command, it may result in the following: Figure 5 As shown, confirmation is indicated by a voice reply of "yes".

[0119] In this embodiment of the application, since the aforementioned steps have confirmed that the current environment is noisy, the voice recognition method may no longer be able to accurately recognize the voice commands of user 001. Therefore, when the voice assistant confirms with user 001, the visual recognition function can be enabled at the same time to facilitate the recognition of confirmation made by user 001 through body movements. For example, user 001 may indicate confirmation by nodding or making an OK gesture.

[0120] The visual recognition function can detect the body movements of user 001. This function can be implemented using the image processor of robot 002. The image processor can acquire images of user 001 and analyze the body movements within those images to obtain visual recognition results. For example, after the image processor acquires an image of user 001 nodding, it can analyze the image and obtain the corresponding text, such as "confirm" or "yes."

[0121] In some embodiments, to further enhance the accuracy of the voice assistant's recognition of user 001's voice commands, the voice assistant can also activate noise detection in addition to visual recognition when confirming user 001's voice commands, in order to detect ambient noise in real time. If the ambient noise is determined to be below a set value, it can be determined that the voice recognition method can recognize user 001's voice commands, and then the voice recognition method can be used to recognize user 001's confirmation command or subsequent voice commands. If the ambient noise is still above the set value, lip reading, visual recognition, or a combination of both methods can be used to recognize user 001's confirmation command or subsequent voice commands.

[0122] S306: Execute the function corresponding to the lip reading result based on the lip reading result.

[0123] In the implementation of this application, if Figure 5 As shown, if the lip reading result is "tell a story", the voice assistant can execute the function corresponding to "tell a story" after user 001 confirms that the lip reading result is correct.

[0124] In some embodiments, during the execution of the task corresponding to the voice command of user 001, the visual recognition function can be continuously enabled to maintain the voice assistant's recognition results of user 001's body movements.

[0125] For example, during the "storytelling" task, lip-reading can be continuously activated to obtain the lip-reading results from user 001. Simultaneously, visual recognition can be activated to maintain the voice assistant's recognition of user 001's body movements. For instance, if user 001 makes a gesture with five fingers spread within the range of robot 002's camera to indicate stopping the storytelling task, the voice assistant can recognize this gesture and stop the task.

[0126] It is understood that in this embodiment of the application, the voice recognition function and the lip reading function are both enabled simultaneously when the visual recognition function is enabled.

[0127] In other embodiments, to further enhance the accuracy of the voice assistant's recognition of user 001's voice commands, in addition to enabling visual recognition, noise detection can also be enabled as described above during the execution of tasks corresponding to user 001's voice commands, so as to detect ambient noise in real time. If the ambient noise is determined to be below a set value, it can be determined that the voice recognition method can already recognize user 001's voice commands, and the voice recognition method can then be used. If the ambient noise is still above the set value, lip reading, visual recognition, or a combination of both methods can be used to accurately recognize other voice commands from user 001.

[0128] For example, during the "storytelling" task, the voice assistant simultaneously activates visual recognition, lip reading, and noise detection. At a certain moment, after detecting that the ambient noise is below a set value, the voice assistant determines that it can recognize the user's voice command using voice recognition. It then switches to voice recognition to recognize the user's voice command during the task. For instance, after the voice assistant has switched to voice recognition to recognize the user's voice command during the task, the user issues the command "stop telling stories." The voice assistant can recognize the "stop telling stories" voice command, obtain the voice recognition result of "stop telling stories," and execute the function corresponding to the voice recognition result of "stop telling stories."

[0129] It is understood that in the embodiments of this application, the speech recognition function and the lip reading function are always enabled during the process of the voice assistant executing the user's 001 voice command.

[0130] S307: Notify user that recognition failed (001).

[0131] In some embodiments, the way to remind the user 001 of recognition failure is to display prompts such as "recognition error" or "unable to recognize" on the screen of robot 002.

[0132] In some embodiments, the way to remind the user of 001 recognition failure can also be by prompting the user with voice messages such as "recognition error" or "unable to recognize".

[0133] In some embodiments, after reminding the user 001 that recognition has failed, the user 001 may be reminded to face the camera or to raise their voice to issue the voice command again.

[0134] S308: Obtain speech recognition results.

[0135] In this embodiment of the application, after obtaining the speech recognition result, the function corresponding to the speech recognition result can be executed based on the speech recognition result.

[0136] In the embodiments of this application, Figure 3 The provided speech recognition method first determines that the voice assistant has been receiving sound for a considerable period of time, suggesting that it may no longer be able to determine when the user's voice command has ended. This indicates an overly noisy external environment, making speech recognition ineffective in such situations. Further, it determines whether lip reading should be used by checking if the user is interacting with the voice assistant. This effectively avoids the reduced accuracy that can occur when lip reading is used when speech recognition is possible, thus improving the accuracy of voice command recognition. Additionally, by re-questioning the user after obtaining the lip reading result, the accuracy of the recognition result can be further guaranteed.

[0137] In this embodiment of the application, the determination of whether user 001 maintains the interaction state with the voice assistant in step 303 can be made from the following aspects:

[0138] First, during the recording process, did the user 001 who interacted with the voice assistant remain unchanged?

[0139] If the voice assistant detects that user 001, who has been interacting with the voice assistant during the recording process, has not changed, then the probability that user 001 and the voice assistant are interacting is relatively high.

[0140] If the user 001 interacting with the voice assistant has changed, the user 001 who issued the voice command may have left. In some implementations, this can be directly determined as invalid. In other implementations, the voice command of the last user 001 who interacted with the voice assistant during the recording process can be detected. For example, if the voice assistant detects that the person interacting with it has changed once during the recording process, meaning that two users 001 interacted with it during the recording process, then the voice command of the second user 001 who interacted with it during the recording process can be detected.

[0141] In this embodiment of the application, face tracking technology can be used to detect whether the user 001 who interacts with the voice assistant has changed.

[0142] In some embodiments, if some electronic devices do not have face tracking technology or face tracking capabilities, they can detect whether the user 001 interacting with the voice assistant has changed by detecting whether the face of the user 001 directly in front of the voice assistant has changed.

[0143] Second, whether the user 001's willingness to interact with the voice assistant reaches the second set value.

[0144] In some embodiments, the interaction willingness value can be calculated based on factors such as the distance between user 001's face and the voice assistant, and the orientation of the user's face over a period of time. For example, if the distance between user 001's face and the voice assistant is relatively close over a period of time, and user 001's face is facing the voice assistant, then the interaction willingness value is higher; otherwise, it is lower.

[0145] Specifically, in one possible implementation, the voice assistant can acquire the facial angle of user 001 and the distance between user 001 and robot 002 by collecting images of user 001 over a period of time. Then, based on the facial angle and distance between user 001 and the smart device, an interaction willingness value model is used to obtain the interaction willingness value of user 001. The higher the interaction willingness value, the stronger the interaction between user 001 and the voice assistant.

[0146] In the interaction willingness value model, different values ​​can be defined for different facial angles and different values ​​for the distance between user 001 and robot 002. Different weights can be assigned to the values ​​corresponding to the facial angles and the distance between user 001 and robot 002. For example, since facial angles are more indicative of whether user 001 is interacting with the voice assistant, the weight corresponding to the facial angles can account for 60%, and the weight corresponding to the distance between user 001 and robot 002 can account for 40%.

[0147] It's understandable that when user 001's willingness to interact with the electronic device is weak, the distance between user 001 and the electronic device is relatively far, and the angle of the user's face deviates to some extent from the angle directly facing the electronic device. In this case, accurately capturing and recognizing user 001's lip movements is difficult, and therefore, lip-reading recognition is insufficient to accurately recognize user 001's voice commands. Conversely, when user 001's willingness to interact with the electronic device is strong, the distance between user 001 and the electronic device is relatively close, and the angle of the user's face is close to or equal to the angle directly facing the electronic device. In this case, the electronic device can accurately capture and recognize user 001's lip movements, and therefore, lip-reading recognition can accurately recognize user 001's voice commands.

[0148] In some embodiments, to more accurately determine whether lip reading is used to obtain the recognition result, the following can be performed: Figure 3 The judgment conditions in steps 302 and 303 are sorted and supplemented. The specific speech recognition method is as follows: Figure 6 As shown, steps 301 and 304-308 are as described above and will not be repeated here. Steps 302-303 will be described in detail below. Specifically, steps 302-303 can be adjusted as follows:

[0149] S302A: Determine whether there is a human voice in the received sound.

[0150] If the judgment result is yes, it indicates that user 001 has issued a voice command, and the process proceeds to S302B; if the judgment result is no, it indicates that user 001 has not issued a voice command, and the process proceeds to S302C to restart the combination of received sound and re-perform the detection.

[0151] In this embodiment, the presence of human voice in the received sound can be detected by a human voice detection model in the artificial intelligence processor. If human voice is present, S302B is executed to further determine whether the duration of sound reception detected by the voice assistant is less than a first preset value. If no human voice is present, the process can proceed to S302C after a preset interval to restart sound reception and recalculate the duration of sound reception. For example, the preset interval can be 200ms.

[0152] S302B: Determine whether the conditions for using voice recognition to recognize the user's voice commands are met;

[0153] If the judgment result is yes, it indicates that the voice recognition method can recognize the user's voice commands, and proceed to S308 to obtain the voice recognition result; if it is not satisfied, it indicates that the voice recognition method can no longer accurately recognize the user's voice commands, and proceed to S303A to check whether the conditions for entering the lip reading recognition mode are met.

[0154] Among them, the conditions for voice command recognition are as follows: Figure 3 The steps described in step S302 will not be repeated here.

[0155] S302C: Restarts sound reception and recalculates the duration of the reception.

[0156] S303A: Determine whether the user 001 being tracked for face has not changed.

[0157] If the judgment result is yes, it indicates that the user interacting with the voice assistant during the sound recording process has been the same user. Then proceed to S303B and use that user as the user interacting with the voice assistant. If the judgment result is no, it indicates that the user interacting with the voice assistant during the sound recording process has been changed. Then proceed to S303C and use the last user 001 captured by the camera device as the user 001 interacting with the voice assistant.

[0158] It is understandable that when determining whether a user is interacting with a voice assistant, it is necessary to first identify the user interacting with the voice assistant. This is because during a recording process, due to the noisy environment, VAD technology may not be able to accurately determine the timing of human voices. Therefore, multiple users may have interacted with the voice assistant during this recording process, meaning that the user interacting with the voice assistant may have changed. For example, during this recording process, two users may have interacted with the voice assistant in turn, while the user before the change may have left. Therefore, the second user after the change can be regarded as the user interacting with the voice assistant, and lip reading can be performed on the mouth features of the second user.

[0159] S303B: Designates the current user as the user interacting with the voice assistant. Here, the current user refers to the user who has been continuously interacting with the voice assistant during the audio recording process.

[0160] S303C: The last user 001 captured by the camera device is used as the user 001 interacting with the voice assistant.

[0161] S303D: Determine whether the interaction intention of user 001 who is interacting with the voice assistant has reached the first preset value.

[0162] If the judgment result is yes, it indicates that user 001 is interacting with the device, and proceeds to S305 to obtain the lip reading result; if the judgment result is no, it indicates that user 001's willingness to interact with the voice assistant is weak, and it is difficult to recognize the user's voice commands using lip reading, and proceeds to step S307 to prompt the user that the recognition failed.

[0163] This application Figure 6 The speech recognition method shown can sort several judgment conditions for using lip reading, enabling more accurate determination of when to use lip reading. Furthermore, it can prematurely end the current process and start the next round of detection if no human voice is detected in the audio recording, avoiding unnecessary subsequent recognition steps and effectively improving recognition efficiency.

[0164] In summary, the speech recognition method provided in this application first determines that the voice assistant has been receiving sound for a considerable period of time, indicating that the voice assistant may no longer be able to determine when the user's voice command has ended. This suggests that the external environment is too noisy, making it difficult to recognize the user's voice command under such circumstances. Furthermore, it determines whether to use lip reading by judging whether the user is interacting with the voice assistant. This effectively avoids the reduced accuracy caused by using lip reading when speech recognition is possible, thus significantly improving the accuracy of voice command recognition.

[0165] In addition, the ability to ask user 001 a second question after obtaining the lip reading results can effectively ensure the accuracy of the recognition results.

[0166] Secondly, in the speech recognition method provided in this application embodiment, visual recognition function and noise detection function can be enabled at the same time as confirming with user 001. On the one hand, the speech assistant can maintain the recognition results of user 001's body movements, and on the other hand, the speech command recognition method can be adjusted in a timely manner according to the change of ambient noise, thereby increasing the accuracy of speech command recognition.

[0167] This application embodiment also provides a voice interaction device, including:

[0168] The detection module is used to control the electronic device to enter the voice recognition mode after detecting that the user 001 has woken up the voice assistant.

[0169] The recognition control module is used to control the electronic device to recognize the voice commands of user 001 and obtain the voice recognition results if the current voice interaction environment of the electronic device meets the voice recognition conditions.

[0170] If the current voice interaction environment of the electronic device does not meet the voice recognition conditions, then it is detected whether the current interaction state of user 001 meets the lip reading recognition conditions. If it is determined that the current interaction state of user 001 meets the lip reading recognition conditions, the electronic device is controlled to use lip reading recognition to identify the mouth change features of user 001 obtained by the electronic device through the image acquisition device to obtain the lip reading recognition result.

[0171] The execution module is used to control the electronic device to perform the functions corresponding to the recognition results obtained by the electronic device. For example, if the electronic device obtains lip reading results, it controls the electronic device to perform the functions corresponding to the lip reading results; if the electronic device obtains speech recognition results, it controls the electronic device to perform the functions corresponding to the speech recognition results.

[0172] The embodiments disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0173] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0174] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0175] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0176] This application also provides a computer program or a computer program product including a computer program, which, when executed on a computer, causes the computer to implement the aforementioned voice command execution method. Alternatively, the computer program product may include instructions for implementing the aforementioned voice interaction method.

[0177] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0178] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0179] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0180] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A voice interaction method applied to electronic devices, characterized in that, The method comprises: In the case where it is detected that the current voice interaction environment of the electronic device does not meet the voice recognition condition, it is determined whether the current interaction state of the user meets the lip-reading condition, wherein the voice recognition condition comprises that the duration of sound collection of the electronic device is greater than zero and less than a set time; wherein the duration of sound collection of the electronic device greater than zero and less than a set time is used to represent the time point at which the electronic device can judge the user's voice truncation; In the case where it is determined that the current interaction state of the user meets the lip-reading condition, the lip-reading result obtained by recognizing the mouth change characteristics of the user obtained by the image acquisition device of the electronic device in the lip-reading mode is acquired; It is confirmed whether the lip-reading result is correct; In the case where it is confirmed that the lip-reading result is correct, the function corresponding to the lip-reading result is executed; The determination that the current interaction state of the user meets the lip-reading condition comprises: It is determined whether the user interacting with the electronic device is replaced within the set time; It is detected whether the interaction intensity value of the user with the electronic device reaches a set intensity value; In the case where it is determined that the user interacting with the electronic device is not replaced within the set time, and the interaction intensity value of the user with the electronic device reaches the set intensity value, it is confirmed that the user and the electronic device are in an interaction state; Or, in the case where it is determined that the user interacting with the electronic device is replaced within the set time, and the interaction intensity value of the last user interacting with the electronic device reaches the set intensity value, it is confirmed that the last user interacting with the electronic device and the electronic device are in an interaction state; The step of confirming whether the lip-reading result is correct also comprises: The limb action characteristics of the user and the noise value around the electronic device are acquired, wherein the limb action characteristics are used to confirm whether the lip-reading result is accurate; When it is determined that the noise value is lower than a set value, the user's instruction is recognized based on the voice recognition mode; When it is determined that the noise value is higher than the set value, the user's instruction is recognized based on the lip-reading mode, the visual recognition mode, or the combination of the lip-reading mode and the visual recognition mode.

2. The method of claim 1, wherein, The voice recognition condition comprises: In the case where the noise value around the electronic device is greater than or equal to a set noise value, the duration of sound collection of the electronic device is greater than zero and less than a set time.

3. The method of claim 1, wherein: The interaction intensity value is related to the distance between the user and the electronic device and the face orientation of the user.

4. The method of claim 1, wherein, The method for confirming whether the lip-reading result is correct comprises: Asking the user whether the function corresponding to the lip-reading result needs to be executed; In the case where the user confirms that the function corresponding to the lip-reading result needs to be executed, it is confirmed that the lip-reading result is correct.

5. The method according to any one of claims 1 to 4, characterized in that, The electronic device is a robot.

6. An electronic device, comprising: Comprise: a memory for storing instructions executed by one or more processors of the electronic device, and The processor is one of the one or more processors of the electronic device and is configured to execute the voice interaction method of any one of claims 1-5.

7. A computer readable storage medium characterized by The computer readable storage medium has stored thereon instructions that, when executed, cause a computer to perform the voice interaction method of any one of claims 1-5.

8. A computer program product, characterised in that, The computer program product includes instructions that, when executed, cause a computer to perform the voice interaction method of any one of claims 1-5.

Images