A wake-up-free voice recognition method, device, equipment and storage medium

Abstract

The present invention discloses a wake-up-free speech recognition method, apparatus, device, and storage medium. The method comprises: collecting a user's visual information through a visual sensor, and determining the gaze start and end time corresponding to a gaze event based on the visual information; collecting the user's voice information through a voice sensor, and determining the voice detection start and end time corresponding to a voice detection event based on the voice information; determining the temporal overlap between the gaze event and the voice detection event based on the gaze start and end time and the voice detection start and end time; and performing voice recognition on the voice information based on the temporal overlap to determine a voice recognition result, thereby implementing a wake-up-free speech recognition method and improving the user experience.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a wake-up-free speech recognition method, apparatus, device, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, voice recognition plays an important role in the control of artificial intelligence (AI) devices such as in-vehicle equipment.

[0003] Currently, before using voice to control AI devices, you usually need to start the voice recognition and control function by saying a wake-up word or turning on the voice function startup interface, button or icon before you can issue voice commands.

[0004] However, if the user is not familiar with the voice function of the AI ​​device, or the voice command is urgent, this is very unfriendly to the user experience and goes against the original intention of voice recognition. Summary of the Invention

[0005] The present invention provides a wake-up-free speech recognition method, apparatus, device and storage medium to solve the problem that the speech recognition function needs to be woken up before speech recognition, realize the wake-up-free speech recognition method, and improve the user experience.

[0006] According to one aspect of the present invention, a wake-up-free speech recognition method is provided, comprising:

[0007] collecting visual information of the user through a visual sensor, and determining the start and end time of the gaze event corresponding to the gaze event based on the visual information;

[0008] Collecting the user's voice information through a voice sensor, and determining the voice detection start and end time corresponding to the voice detection event according to the voice information;

[0009] Determining the temporal overlap between the gaze event and the voice detection event according to the gaze start and end times and the voice detection start and end times;

[0010] The voice information is subjected to voice recognition according to the time overlap to determine a voice recognition result.

[0011] According to another aspect of the present invention, a wake-up-free speech recognition device is provided, characterized by comprising:

[0012] A visual recognition module, configured to collect visual information of the user through a visual sensor, and determine the start and end time of a gaze event according to the visual information;

[0013] A voice detection module is used to collect the user's voice information through a voice sensor and determine the voice detection start and end time corresponding to the voice detection event based on the voice information;

[0014] A coincidence determination module, configured to determine a temporal coincidence between the gaze event and the voice detection event according to the gaze start and end time and the voice detection start and end time;

[0015] A recognition result determination module is used to perform voice recognition on the voice information according to the time overlap to determine a voice recognition result.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the wake-up-free speech recognition method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the wake-up-free speech recognition method described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention collects the user's visual information through a visual sensor, and determines the line of sight start and end time corresponding to the line of sight event based on the visual information; collects the user's voice information through a voice sensor, and determines the voice detection start and end time corresponding to the voice detection event based on the voice information; determines the time overlap between the line of sight event and the voice detection event based on the line of sight start and end time and the voice detection start and end time; performs voice recognition on the voice information based on the time overlap to determine the voice recognition result, which solves the problem of needing to wake up the voice recognition function before voice recognition, achieves the beneficial effect of realizing a wake-up-free voice recognition method and improving the user experience.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a flowchart of a wake-up-free speech recognition method provided according to the first embodiment of the present invention;

[0025] Figure 2 This is a flowchart of a wake-up-free speech recognition method provided according to the second embodiment of the present invention;

[0026] Figures 3A to 3C is a schematic diagram of the temporal overlap of gaze fixation events and speech detection events;

[0027] Figure 4 This is a flowchart of another wake-up-free speech recognition method provided according to the second embodiment of the present invention;

[0028] Figure 5 2 is a schematic structural diagram of a wake-up-free speech recognition device provided according to Embodiment 3 of the present invention;

[0029] Figure 6 The figure is a structural diagram of an electronic device for implementing the wake-up-free speech recognition method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Example 1

[0033] Figure 1 A flowchart of a wake-up-free voice recognition method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where the voice recognition function of an electronic device is activated without wake-up and the voice uttered by the user is recognized. The method can be executed by a wake-up-free voice recognition device. The wake-up-free voice recognition device can be implemented in the form of hardware and / or software. The wake-up-free voice recognition device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0034] S110 , collecting visual information of the user through a visual sensor, and determining the gaze start and end time corresponding to the gaze event according to the visual information.

[0035] The visual sensor may be any device with an image acquisition function, such as an image collector, etc. Visual information is information collected by the visual sensor that can reflect the area where the user's line of sight falls, and may include, for example, the area where the user's line of sight falls and the angle of sight, etc.

[0036] It should be noted that the information collected by the visual sensor may include not only visual information but also other facial information, gesture information, or behavioral information. Therefore, a series of preprocessing operations such as detection, recognition, classification, screening, and filtering are required to determine the visual information collected by the visual sensor. The specific methods of the above preprocessing operations are not limited in the embodiments of the present invention.

[0037] Among them, gaze events refer to events related to user gaze recognition, such as the gaze start event and gaze end event corresponding to when the user's gaze angle is the target angle (that is, the user's gaze is fixed on the target area). The gaze start event can be considered as the event when the user's gaze starts to fixate on the target area; the gaze end event can be considered as the event when the user ends fixing on the target area. The target area can be the area where the control object corresponding to the voice information is located, or it can be a preset area, such as the central control screen of the voice recognition device.

[0038] Specifically, visual recognition is performed on the user's visual information collected by the visual sensor to determine the gaze events, that is, the gaze start event and the gaze end event of the user's gaze on the target area, and the gaze start and end times corresponding to the gaze start event and the gaze end event are obtained.

[0039] S120: Collect the user's voice information through the voice sensor, and determine the voice detection start and end time corresponding to the voice detection event according to the voice information.

[0040] The voice sensor can be any device with a sound collection function, such as a microphone, etc. Voice information is the voice command information issued by the user collected by the visual sensor, such as "Call Wang" or "Increase the air conditioning temperature, etc."

[0041] It should be noted that the voice information collected by the voice sensor includes not only the user's voice commands but also ambient noise or periods of blank audio. Therefore, the voice information collected by the voice sensor requires a series of preprocessing operations, such as denoising, screening, and filtering, to determine the voice information. The voice information can also be classified to determine the control object corresponding to the voice information. The embodiments of the present invention do not limit the specific methods of these preprocessing operations.

[0042] A voice detection event refers to an event that detects the audio of a user's voice. For example, it may include a voice detection start event and a voice detection end event. A voice detection start event can be considered an event in which a user begins to speak, while a voice detection end event can be considered an event in which a user stops speaking.

[0043] Specifically, voice detection is performed on the user's voice information collected by the voice sensor, and voice detection events, namely, a voice detection start event and a voice detection end event, are determined, and the voice detection start and end times corresponding to the voice detection start event and the voice detection end event are obtained.

[0044] S130 , determining the temporal overlap between the gaze event and the voice detection event according to the gaze start and end times and the voice detection start and end times.

[0045] The temporal overlap between gaze events and voice detection events indicates the temporal overlap between the user issuing a voice command and their gaze directed toward the target area. For example, if the user speaks "Open the left window" and gazes toward the left window, then if the two events overlap in time, it means that the user was also gazed toward the left window while speaking "Open the left window."

[0046] Specifically, based on the gaze start and end time corresponding to the gaze event and the voice detection start and end time corresponding to the voice detection event, the time overlap of the gaze event and the voice detection event can be determined, which is used to reflect whether the user's voice and gaze are synchronized.

[0047] S140: Perform speech recognition on the speech information according to the time overlap to determine a speech recognition result.

[0048] Specifically, if the temporal overlap between the gaze event and the voice detection event reaches a certain threshold, it can be assumed that the voice information emitted by the user is consistent with the target area of ​​the gaze, and the voice information emitted by the user is determined to be voice information that needs to be recognized. Therefore, the voice recognition function is enabled and the voice information is subjected to voice recognition to determine the voice recognition result. If the temporal overlap between the gaze event and the voice detection event does not reach a certain threshold, it can be assumed that the voice information emitted by the user is inconsistent with the target area of ​​the gaze, and the voice information emitted by the user is not voice information that needs to be recognized, so it can remain silent and no feedback is provided.

[0049] Therefore, when the user needs to issue a voice command, there is no need to issue a preset wake-up command first. The user only needs to look at the control object of the voice command while issuing the voice command to achieve wake-up-free voice recognition.

[0050] It should be noted that the wake-up-free voice recognition method provided in the embodiments of the present invention can be used in combination with any one or more existing voice recognition methods to enrich the voice recognition capabilities of the device, meet the needs of users in different usage scenarios, and enhance the user experience of voice recognition control. The scenarios in which the wake-up-free voice recognition method provided in the embodiments of the present invention can be applied include, but are not limited to, controlling in-vehicle devices, and can also be used to control smart homes or smart terminal devices.

[0051] In an embodiment of the present invention, visual information of a user is collected through a visual sensor, and the start and end time of a gaze event corresponding to the gaze event is determined based on the visual information; voice information of the user is collected through a voice sensor, and the start and end time of a voice detection event is determined based on the voice information; the time overlap between the gaze event and the voice detection event is determined based on the gaze start and end time and the voice detection start and end time; voice recognition is performed on the voice information based on the time overlap to determine a voice recognition result, thereby solving the problem of needing to wake up the voice recognition function before voice recognition, realizing a wake-up-free voice recognition method based on visual information and voice information, thereby enhancing the convenience of voice control and improving the user experience.

[0052] Optionally, determining the gaze start and end time corresponding to the gaze event according to the visual information in step S110 includes:

[0053] S111, performing visual recognition on the visual information to determine gaze events, where the gaze events include: gaze start events and gaze end events corresponding to when the user's gaze angle is a target angle, where the target angle is the angle of the control object corresponding to the voice information;

[0054] The user sight angle is the angle between the user's sight and the baseline. The user sight angle can be used to indicate the area of ​​focus. The target angle is the angle between the execution object corresponding to the voice information and the base line. The target angle can be a specific value or a preset range. The target angle of each control object can be determined based on the actual scenario and is not limited in this embodiment of the present invention.

[0055] For example, in a vehicle cockpit scenario, the direction of the driver's seat facing the right window is set as the baseline. If the voice information is classified and determined to be controlled as the left window, the target angle is 180 degrees; if the voice information is classified and determined to be controlled as the vehicle host, the target angle is 15 degrees to 30 degrees.

[0056] Specifically, visual recognition is performed on the visual information to determine the user's line of sight angle. If the user's line of sight angle is the target angle corresponding to the control object corresponding to the voice information, it is considered that the user is looking at the control object corresponding to the voice information, and the event that the user starts looking at the control object is determined as a gaze start event, and the event that the user ends looking at the control object is determined as a gaze end event.

[0057] S112: Obtain the gaze start time corresponding to the gaze start event and the gaze end time corresponding to the gaze end event.

[0058] Specifically, the gaze start time corresponding to the gaze start event formed when the user starts gazing at the control object and the gaze end time corresponding to the gaze end event formed when the user stops gazing at the control object are obtained. The time period during which the user gazes at the control object can be determined based on the gaze start time and the gaze end time.

[0059] Optionally, determining the voice detection start and end time corresponding to the voice detection event according to the voice information in step S120 includes:

[0060] S121: Perform voice detection on the voice information to determine a voice detection event. The voice detection event includes: a voice detection start event and a voice detection end event.

[0061] Specifically, when a voice message from a user is received, the user voice message is detected, and the event of detecting that the user starts to send a voice message is determined as a voice detection start event, and the event of detecting that the user ends sending a voice message for a preset time is determined as a voice detection end event.

[0062] S122: Obtain the voice detection start time corresponding to the voice detection start event and the voice detection end time corresponding to the voice detection end event.

[0063] Specifically, the voice detection start time corresponding to the voice detection start event (when the user begins speaking a voice message) and the voice detection end time corresponding to the voice detection end event (when the user ends speaking a voice message) are obtained. The time period during which the user spoke can be determined based on the voice detection start and end times.

[0064] Example 2

[0065] Figure 2 This is a flowchart of a wake-up-free speech recognition method provided by the second embodiment of the present invention. This embodiment further refines steps S130 and S140 of the above embodiment. Figure 2 As shown, the method includes:

[0066] S210 , collecting visual information of the user through a visual sensor, and determining the gaze start and end time corresponding to the gaze event according to the visual information.

[0067] S220: Collect the user's voice information through the voice sensor, and determine the voice detection start and end time corresponding to the voice detection event according to the voice information.

[0068] S230: Determine the overlapping time length of the gaze event and the voice detection event according to the gaze start and end time and the voice detection start and end time.

[0069] For example, Figure 3AAs shown, the start and end times of the line of sight fixation are Te1 to Te2, and the start and end times of the voice detection are Tv1 to Tv2: Te1 < Tv1 < Tv2 < Te2. Then, the overlapping time length of the line of sight fixation event and the voice detection event is Tv2 - Tv1. As Figure 3B shown, the start and end times of the line of sight fixation are Te3 to Te4, and the start and end times of the voice detection are Tv1 to Tv2: Tv1 < Te3 < Te4 < Tv2. Then, the overlapping time length of the line of sight fixation event and the voice detection event is Te4 - Te3. As Figure 3C shown, the start and end times of the line of sight fixation are Te5 to Te6 and Te7 to Te8, and the start and end times of the voice detection are Tv1 to Tv2: Te5 < Tv1 < Te6 < Te7 < Tv2 < Te8. Then, the overlapping time length of the line of sight fixation event and the voice detection event is (Te6 - Tv1) + (Tv2 - Te7).

[0070] S240. Determine the voice time length according to the start and end times of the voice detection, and determine the ratio of the overlapping time length to the voice time length as the time overlapping degree.

[0071] Exemplarily, as Figure 3A shown to Figure 3B shown, the voice time length is the difference between the voice detection end time and the voice detection start time, that is, Tv2 - Tv1. For continuous line of sight fixation without interruption, Figure 3A the time overlapping degree of the line of sight fixation event and the voice detection event shown is: (Tv2 - Tv1) / (Tv2 - Tv1) = 100%; <s Figure 3B the time overlapping degree of the line of sight fixation event and the voice detection event shown is: (Tv4 - Tv3) / (Tv2 - Tv1). For continuous line of sight fixation with interruption, Figure 3C the time overlapping degree of the line of sight fixation event and the voice detection event shown is ((Te6 - Tv1) + (Tv2 - Te7)) / (Tv2 - Tv1).

[0072] S250. If the time overlapping degree is greater than the first preset threshold, perform voice recognition on the voice information to determine the voice recognition result.

[0073] Among them, the first preset threshold can be understood as the expected time overlapping degree, which can be set according to actual needs or by the user according to the actual usage scenario. The embodiments of the present invention do not limit this.

[0074] Specifically, if the temporal overlap exceeds a first preset threshold, it indicates that the user's voice message and the controlled object being gazed at are consistent, indicating that the user's voice message needs to be recognized. Therefore, voice recognition is performed on the voice message to determine a voice recognition result. The voice recognition result can be the user's semantic intent or semantic recognition feedback information, such as "Unclear user intent, please repeat."

[0075] For example, the first preset threshold is set to 80%, then Figure 3A The temporal overlap between the gaze event and the voice detection event is 100%, and voice recognition is performed on the voice information to determine a voice recognition result. Figure 3B If the temporal overlap between the gaze event and the voice detection event is 70%, then there is no need to perform voice recognition on the voice information, and the system remains silent, waiting for the input of voice information and visual information.

[0076] In an embodiment of the present invention, visual information of a user is collected through a visual sensor, and the line of sight start and end time corresponding to the line of sight event is determined based on the visual information; voice information of the user is collected through a voice sensor, and the voice detection start and end time corresponding to the voice detection event is determined based on the voice information; the overlapping time length of the line of sight event and the voice detection event is determined based on the line of sight start and end time and the voice detection start and end time; the voice time length is determined based on the voice detection start and end time; the ratio of the overlapping time length to the voice time length is determined as the time overlap; if the time overlap is greater than a first preset threshold, voice recognition is performed on the voice information to determine a voice recognition result, thereby realizing a wake-up-free voice recognition method based on visual information and voice information, enhancing the convenience of voice control, and improving the user experience.

[0077] Optionally, step S230 includes: performing speech recognition on the speech information to determine a speech recognition result, including:

[0078] S231. Perform speech recognition on the speech information to determine the semantic intent, and determine the semantic understanding index of the semantic intent.

[0079] Among them, the semantic understanding index can be understood as the certainty of the semantic intention determined after voice recognition of the user's voice information. The value range of the semantic understanding index can be 0 to 1. The higher the index, the more accurate the semantic understanding, and vice versa.

[0080] Specifically, any semantic recognition method can be used to identify the semantic intent of the speech information, and the embodiments of the present invention do not impose any restrictions on this. For example, semantic recognition can be performed using a pre-established semantic recognition model, or deep learning can be used for iterative training and recognition. After determining the semantic intent corresponding to the speech information, a semantic understanding index is determined based on the identified semantic intent. The embodiments of the present invention do not impose any restrictions on the method for determining the semantic understanding index.

[0081] For example, if the determined semantic intent is unique, the semantic understanding index is 1; if the determined semantic intent is N, the semantic understanding index is 1 / N.

[0082] For example, if the voice message sent by the user is "go to place A", the semantic intention is "navigate to place A". If only place A is found in the query, the semantic understanding index is 1.

[0083] S232. If the semantic understanding index is greater than or equal to the second preset threshold, the semantic intention is determined as the speech recognition result; if the semantic understanding index is less than the second preset threshold, the preset information corresponding to the semantic understanding index is used as the speech recognition result.

[0084] The second preset threshold can be set according to needs, and the embodiment of the present invention does not impose any limitation on this.

[0085] Specifically, if the semantic understanding index is greater than or equal to a second preset threshold, the recognized semantic intent can be considered to be the accurate intent corresponding to the user's voice, and thus, the semantic intent is determined as the voice recognition result. If the semantic understanding index is less than the second preset threshold, the recognized semantic intent can be considered to be unclear, and thus the preset information corresponding to the semantic understanding index can be used as the voice recognition result.

[0086] For example, the preset information corresponding to the semantic understanding index can be based on the original configuration. For example, if the semantic understanding index is 0 to 0.3, the preset information is "I didn't hear the user's instructions clearly, please say it again"; if the semantic understanding index is 0.3 to 0.7 (the second threshold is 0.7), the preset information is to ask the user whether the intention is the most frequently used intention among all intentions or the most recent historical intention.

[0087] Optionally, after performing speech recognition on the speech information to determine a speech recognition result, the method further includes:

[0088] If the speech recognition result is a semantic intention, the control instruction is determined according to the semantic intention;

[0089] Execute corresponding control operations according to the control instructions.

[0090] Specifically, after performing voice recognition on the voice information based on the voice information and visual information to determine the semantic intention of the voice information sent by the user, the control instruction is determined according to the semantic intention, and the corresponding control operation is performed according to the control instruction to realize wake-up-free voice control.

[0091] like Figure 4 As shown, the specific steps of the embodiment of the present invention include: collecting the user's visual information through a visual sensor and collecting the user's voice information through a voice sensor; determining a gaze event based on the visual information and determining a voice detection event based on the voice information; judging whether the gaze angle in the gaze event is the angle of the execution object corresponding to the voice information, and if so, obtaining the gaze start and end time corresponding to the gaze event and the voice detection start and end time corresponding to the voice detection event; if not, not performing any operation. After obtaining the gaze start and end time and the voice detection start and end time, determining the time overlap of the gaze event and the voice detection event, and further judging whether the time overlap is greater than a first set threshold, if so, enabling wake-up-free voice recognition, performing voice recognition on the voice information to determine the semantic intent, and determining the semantic understanding index of the semantic intent; if not, not performing any operation. After determining the semantic understanding index, judging whether the semantic understanding index is greater than or equal to a second preset threshold, if so, determining the control instruction corresponding to the semantic intent and executing the control operation corresponding to the control instruction to achieve wake-up-free voice control; if not, not performing any operation.

[0092] Example 3

[0093] Figure 5 This is a structural diagram of a wake-up-free speech recognition device provided by the third embodiment of the present invention. Figure 5 As shown, the apparatus includes: a visual recognition module 310, a voice detection module 320, a coincidence determination module 330 and a recognition result determination module 340;

[0094] The visual recognition module 310 is configured to collect visual information of the user through a visual sensor and determine the start and end time of the gaze event according to the visual information;

[0095] The voice detection module 320 is used to collect the user's voice information through the voice sensor and determine the voice detection start and end time corresponding to the voice detection event according to the voice information;

[0096] A coincidence determination module 330 is configured to determine a temporal coincidence between the gaze event and the voice detection event based on the gaze start and end time and the voice detection start and end time;

[0097] The recognition result determination module 340 is configured to perform speech recognition on the speech information according to the time overlap to determine a speech recognition result.

[0098] Optionally, the visual recognition module 310 is specifically configured to:

[0099] Performing visual recognition on the visual information to determine gaze events, the gaze events including: a gaze start event and a gaze end event corresponding to when the user's gaze angle is a target angle; the target angle is the angle of the control object corresponding to the voice information;

[0100] Obtain the gaze start time corresponding to the gaze start event and the gaze end time corresponding to the gaze end event.

[0101] Optionally, the voice detection module 320 is specifically configured to:

[0102] Performing voice detection on the voice information to determine a voice detection event, wherein the voice detection event includes: a voice detection start event and a voice detection end event;

[0103] The voice detection start time corresponding to the voice detection start event and the voice detection end time corresponding to the voice detection end event are obtained.

[0104] Optionally, the overlap determination module 330 is specifically configured to:

[0105] Determining the overlapping time length of the gaze event and the voice detection event according to the gaze start and end time and the voice detection start and end time;

[0106] Determine the duration of the speech according to the start and end time of the speech detection;

[0107] The ratio of the overlap time length to the speech time length is determined as the time overlap degree.

[0108] Optionally, the recognition result determination module 340 is specifically configured to:

[0109] If the time overlap is greater than a first preset threshold, speech recognition is performed on the speech information to determine a speech recognition result.

[0110] Optionally, the recognition result determination module 340 includes:

[0111] a semantic intent determination unit, configured to perform speech recognition on the speech information to determine the semantic intent;

[0112] an understanding index determining unit, configured to determine a semantic understanding index of the semantic intent;

[0113] a first result determination module, configured to determine the semantic intention as the speech recognition result if the semantic understanding index is greater than or equal to a second preset threshold;

[0114] The second result determination module uses the preset information corresponding to the semantic understanding index as the speech recognition result if the semantic understanding index is less than the second preset threshold.

[0115] Optionally, also include:

[0116] an instruction determination module for, after performing speech recognition on the speech information to determine a speech recognition result, determining a control instruction according to the semantic intent if the speech recognition result is the semantic intent;

[0117] An execution module is used to execute corresponding control operations according to the control instructions.

[0118] The wake-up-free speech recognition device provided in the embodiment of the present invention can execute the wake-up-free speech recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0119] Example 4

[0120] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0121] like Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0122] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0123] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the wake-up-free speech recognition method.

[0124] In some embodiments, the wake-up-free speech recognition method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the wake-up-free speech recognition method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the wake-up-free speech recognition method in any other appropriate manner (e.g., by means of firmware).

[0125] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0130] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0131] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0132] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A wake-up-free speech recognition method, characterized in that: include: collecting visual information of the user through a visual sensor, and determining the start and end time of the gaze event corresponding to the gaze event based on the visual information; The gaze events include: gaze start events and gaze end events corresponding to when the user's gaze angle is a target angle; the target angle is the angle of the control object corresponding to the voice information; Collecting the user's voice information through a voice sensor, and determining the voice detection start and end time corresponding to the voice detection event according to the voice information; Determining the temporal overlap between the gaze event and the voice detection event according to the gaze start and end times and the voice detection start and end times; If the time overlap is greater than a first preset threshold, performing speech recognition on the speech information to determine a speech recognition result; The determining of the temporal overlap between the gaze event and the voice detection event according to the gaze start and end time and the voice detection start and end time includes: Determining the overlapping time length of the gaze event and the voice detection event according to the gaze start and end time and the voice detection start and end time; Determine the duration of the speech according to the start and end time of the speech detection; The ratio of the overlap time length to the speech time length is determined as the time overlap degree.

2. The method according to claim 1, characterized in that The determining, according to the visual information, the gaze start and end time corresponding to the gaze event includes: Performing visual recognition on the visual information to determine a gaze event; Obtain the gaze start time corresponding to the gaze start event and the gaze end time corresponding to the gaze end event.

3. The method according to claim 1, characterized in that The determining, according to the voice information, the voice detection start and end time corresponding to the voice detection event includes: Performing voice detection on the voice information to determine a voice detection event, wherein the voice detection event includes: a voice detection start event and a voice detection end event; The voice detection start time corresponding to the voice detection start event and the voice detection end time corresponding to the voice detection end event are obtained.

4. The method according to claim 1, wherein Performing speech recognition on the speech information to determine a speech recognition result includes: Performing speech recognition on the speech information to determine semantic intent; Determine a semantic understanding index of the semantic intent; if the number of determined semantic intents is N, the semantic understanding index is 1 / N; If the semantic understanding index is equal to 1, the semantic intention is determined as the speech recognition result; If the semantic understanding index is less than 1, the preset information corresponding to the semantic understanding index is used as the speech recognition result.

5. The method according to claim 4, characterized in that After performing speech recognition on the speech information to determine a speech recognition result, the method further includes: If the speech recognition result is the semantic intention, determining a control instruction according to the semantic intention; Execute corresponding control operations according to the control instructions.

6. A wake-up-free speech recognition device, characterized in that: include: A visual recognition module, configured to collect visual information of the user through a visual sensor, and determine the start and end time of a gaze event according to the visual information; The gaze events include: gaze start events and gaze end events corresponding to when the user's gaze angle is a target angle; the target angle is the angle of the control object corresponding to the voice information; A voice detection module is used to collect the user's voice information through a voice sensor and determine the voice detection start and end time corresponding to the voice detection event based on the voice information; A coincidence determination module, configured to determine a temporal coincidence between the gaze event and the voice detection event according to the gaze start and end time and the voice detection start and end time; a recognition result determination module, configured to perform speech recognition on the speech information to determine a speech recognition result if the temporal overlap is greater than a first preset threshold; The overlap determination module is specifically configured to: Determining the overlapping time length of the gaze event and the voice detection event according to the gaze start and end time and the voice detection start and end time; Determine the duration of the speech according to the start and end time of the speech detection; The ratio of the overlap time length to the speech time length is determined as the time overlap degree.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so as to enable the at least one processor to perform the wake-up-free speech recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the wake-up-free speech recognition method according to any one of claims 1 to 5 when executed.

Images