Voice instruction recognition method, device and equipment and automatic driving vehicle

Abstract

The invention provides a voice instruction recognition method, device and equipment and an automatic driving vehicle, and relates to the field of automatic driving, in particular to the technical field of vehicle-mounted equipment voice control. According to the implementation scheme, multiple face images of a user are obtained when the user makes a voice; based on the analysis of the mouth shapes in the plurality of face images, whether the user is in a state of issuing a voice instruction during the voice making period is determined, and the analysis of the mouth shapes in the plurality of face images comprises the step of determining whether the mouth shapes of the user during the voice making period accord with speaking features; determining that the user is in a voice instruction issuing state in response to determining that the mouth shape of the user conforms to the speaking feature during the voice making period; acquiring an audio corresponding to the voice; and identifying the audio to determine the content of the voice instruction.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to the field of voice control technology for vehicle-mounted equipment, and specifically to a method, device, electronic device, computer-readable storage medium, and computer program product for voice command recognition. Background Art

[0002] In the field of voice control technology for in-vehicle devices, voice recognition applications no longer just serve as remote controls for in-vehicle devices, but also as friends who accompany users emotionally. With the development of vehicle engineering and voice recognition technology, users have put forward higher requirements for the form and accuracy of communication between people and machines. How to recognize voice commands more accurately has always been a hot topic in this field.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product for voice command recognition.

[0005] According to one aspect of the present disclosure, a voice command recognition method is provided, including acquiring multiple facial images of a user during a period when the user utters a voice; determining whether the user is in a state of issuing a voice command during the period when the user utters a voice based on an analysis of the mouth shapes in the multiple facial images, wherein the analysis of the mouth shapes in the multiple facial images includes determining whether the mouth shapes of the user during the period when the user utters the voice meet the speaking characteristics; in response to determining that the mouth shapes of the user during the period when the user utters the voice meet the speaking characteristics, determining that the user is in a state of issuing a voice command; acquiring audio corresponding to the voice; and recognizing the audio to determine the content of the voice command.

[0006] According to another aspect of the present disclosure, a voice command recognition device is provided, including: an image acquisition module, configured to acquire multiple face images of a user during the period when the user utters a voice; a state judgment module, configured to determine whether the user is in a state of issuing a voice command during the period when the user utters a voice based on an analysis of the mouth shape in the multiple face images, wherein the state determination module includes a speech feature comparison module, configured to determine whether the mouth shape of the user during the period when the user utters the voice meets the speech feature; the state determination module is configured to determine that the user is in a state of issuing a voice command in response to determining that the mouth shape of the user during the period when the user utters the voice meets the speech feature; an audio acquisition module, configured to acquire audio corresponding to the voice; and a voice recognition module, configured to recognize the audio to determine the content of the voice command.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided above in the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided above in the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the method provided above when executed by a processor.

[0010] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising a controller, wherein the controller is configured to execute the method provided above in the present disclosure.

[0011] According to one or more embodiments of the present disclosure, voice instructions can be recognized more accurately.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0015] Figure 2 A flow chart of a voice command recognition method according to an embodiment of the present disclosure is shown;

[0016] Figure 3 A schematic diagram showing a method of determining whether a mouth shape meets a speaking feature according to an embodiment of the present disclosure is shown;

[0017] Figure 4 A schematic diagram showing a height difference variation curve according to an embodiment of the present disclosure is shown;

[0018] Figure 5 A schematic diagram showing a method of determining whether a face of a person is facing a device according to an embodiment of the present disclosure is shown;

[0019] Figure 6 A schematic diagram showing whether a user is looking directly at a device according to an embodiment of the present disclosure is shown;

[0020] Figure 7 A schematic diagram showing a method for voice command recognition according to an embodiment of the present disclosure is shown;

[0021] Figure 8 A structural block diagram of a voice command recognition device according to an embodiment of the present disclosure is shown;

[0022] Fig. 9 A structural block diagram of a voice command recognition device according to another embodiment of the present disclosure is shown;

[0023] Fig.10 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0026] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0027] In the field of voice control technology for in-vehicle devices, voice recognition applications no longer just serve as remote controls for in-vehicle devices, but also as emotional companions for users. With the development of vehicle engineering and voice recognition technology, users have put forward higher requirements for the form and accuracy of communication between people and machines.

[0028] In the related art, in order to control the vehicle-mounted equipment by voice, some specific voice wake-up methods are often required, including semantic wake-up, gesture wake-up, specific word wake-up, etc. However, these methods are very susceptible to external noise or surrounding objects, resulting in frequent false triggering or trigger failure. Therefore, a method that can more accurately recognize voice commands is urgently needed.

[0029] In view of the above technical problems, according to one aspect of the present disclosure, a voice command recognition method is provided.

[0030] Before describing in detail the voice command recognition method according to the embodiment of the present disclosure, Figure 1 Schematic diagram depicting an exemplary system in which the various methods and apparatus described herein may be implemented.

[0031] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0032] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0033] The server 120 may run one or more services or software applications that enable the method of voice command recognition. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. Figure 1In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0034] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0035] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0036] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0037] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.

[0038] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0039] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0040] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0041] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0042] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.

[0043] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0044] The following describes in detail the voice command recognition method according to an embodiment of the present disclosure.

[0045] Figure 2 FIG. 2 shows a flow chart of a method 200 for recognizing a voice command according to an embodiment of the present disclosure. Figure 2 As shown, method 200 includes steps S201, S202, S203, S204 and S205.

[0046] In step S201, a plurality of facial images of a user are acquired while the user is uttering a voice.

[0047] In step S202, based on the analysis of the mouth shapes in the multiple face images, it is determined whether the user is in a state of issuing a voice command during the speech. The analysis of the mouth shapes in the face images includes determining whether the mouth shape of the user during the speech conforms to the speaking characteristics.

[0048] In step S203, in response to determining that the mouth shape of the user during the speech is consistent with the speaking characteristics, it is determined that the user is in a state of issuing a voice command.

[0049] In step S204, audio corresponding to the speech is acquired.

[0050] In step S205, the audio is recognized to determine the content of the voice instruction.

[0051] It should be noted that in the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of information related to images and sounds are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0052] In the example, we can use Figure 1 The sensor 111 shown (such as a visual camera) is used to obtain the face image and audio of the user during the speech. The visual camera can be installed in the car and face the driver's seat or other seats in the car.

[0053] In the example, the acquired facial image may include a complete face or a partial face including the mouth of the face. When the user makes a speech (i.e., speaks), the mouth will show continuous dynamic changes. Such dynamic changes conform to certain characteristics, which are different from the characteristics of the user when coughing, yawning, eating, etc., so these unique characteristics in the dynamic changes of the mouth indicating that the user is speaking can be used as speaking features. When the mouth shape in the facial image conforms to the speaking features, it can be concluded that the user is speaking.

[0054] When it is determined that the user is speaking, that is, the user is in a state of issuing a voice command, the audio corresponding to the voice can be obtained for voice recognition. The audio can be obtained synchronously with the face image, and the device for obtaining the audio can be, for example, a miniature microphone. The synchronously obtained audio can then be stored as a recording file for call and recognition. If it is determined that the user is in a state of issuing a voice command, the corresponding recording file can be called from the storage unit, and the audio in the recording file can be voice recognized to obtain the instruction content that can be understood by the vehicle-mounted electromechanical device. If it is determined that the user is not in a state of issuing a voice command, it means that the audio in the corresponding recording file does not contain the voice command content and can be regarded as useless audio. Useless audio and audio that has been completed by voice recognition will generally not be used again in the future, so these audios that will no longer be used can be deleted periodically to avoid them taking up storage space.

[0055] In the example, after the audio is recognized and the command content is obtained, the obtained command content can be transmitted to the vehicle-mounted electromechanical devices such as doors, windows, and vehicle audio. These vehicle-mounted electromechanical devices can perform corresponding tasks according to the received command content, such as closing the door, turning down the volume of the vehicle audio, etc.

[0056] According to the voice command recognition method of the embodiment of the present disclosure, by analyzing the mouth shape in the face image during the user's voice to determine whether the user's mouth shape during this period meets the speaking characteristics, it is determined whether the user is in a state of issuing a voice command, and the feature analysis of the mouth shape in the face image can be combined to determine the wake-up timing of the voice recognition application. By obtaining audio and performing voice recognition on it only when it is determined that the user is in a state of issuing a voice command, the probability of the voice recognition application being affected by external noise, thereby erroneously triggering the voice recognition application or the voice recognition application failing to trigger, can be reduced. At this time, the accuracy of the recognized command content can also be improved, and the tasks performed by the on-board electromechanical equipment based on the command content can be more in line with the user's expectations, giving the user a better experience.

[0057] The following further describes various aspects of the voice instruction recognition method according to the embodiments of the present disclosure.

[0058] According to some embodiments, when determining whether the mouth shape of the user during speech production meets the speech characteristics, multiple height differences formed between the lips of the user during speech production can be determined based on multiple face images. Then, it can be determined whether there is an effective height difference greater than or equal to a predetermined height threshold among these height differences. When it is determined that there is no effective height difference among these height differences, it can be determined that the mouth shape of the user during speech production does not meet the speech characteristics.

[0059] In an example, the height difference between the lowest point of the bottom border of the upper lip and the highest point of the top border of the lower lip may be taken as the height difference formed between the lips.

[0060] Figure 3 FIG. 1 is a schematic diagram showing a method for determining whether a mouth shape conforms to a speech feature according to an embodiment of the present disclosure. Figure 3 Three mouth shapes 310 , 320 , and 330 are shown, and a height threshold 301 is shown.

[0061] In the example, mouth shape 310 is in a closed state, the upper lip and the lower lip are attached, and there is no height difference between the lips, or in other words, the height difference between the lips is 0. Therefore, it can be determined that the height difference between the lips of mouth shape 310 is not a valid height difference. Mouth shape 320 is in an open state, the upper lip and the lower lip are separated, there is a height difference 321 between the lips, and the height difference 321 of mouth shape 320 is greater than the height threshold 301, so it can be determined that the height difference 321 of mouth shape 320 is a valid height difference. However, not all height differences of mouth shapes in an open state are valid height differences. For example, see Figure 3 The mouth shape 330 is also in an open state, and there is a height difference 331 between the upper lip and the lower lip, but the height difference 331 of the mouth shape 330 is less than the height threshold 301, so it can be determined that the height difference 331 of the mouth shape 330 is not a valid height difference.

[0062] Assuming that the mouth shape of the user during speech includes mouth shapes 310, 320 and 330, since the height difference 321 is greater than the height threshold 301, it is a valid height difference, so it can be considered that the mouth shape of the user during speech is consistent with the speech characteristics. Assuming that there is no valid height difference greater than or equal to the height threshold 301 in the mouth shape of the user during speech, it can be determined that the mouth shape of the user during speech does not meet the speech characteristics, and further it can be determined that the user is not in a state of issuing a voice command.

[0063] According to an embodiment of the present disclosure, by judging whether there is an effective height difference between the lips, it is possible to determine whether the user's mouth shape conforms to the speaking characteristics and whether the user is in a state of issuing voice commands based on the degree of lip opening, thereby improving the accuracy of determining the timing for voice recognition of audio.

[0064] According to some embodiments, when determining whether the mouth shape of the user during speech production meets the speech characteristics, the number of peak height differences that characterize the maximum opening degree of the lips in the effective height differences can also be determined in response to determining that there is a valid height difference among multiple height differences. It can then be determined whether the number of peak height differences is greater than or equal to a predetermined number threshold. When it is determined that the number of peak height differences is less than the predetermined number threshold, it can be determined that the mouth shape of the user during speech production does not meet the speech characteristics.

[0065] When issuing a voice command, the user generally mentions the name of the vehicle-mounted electromechanical device to be controlled and the task or action that the device is expected to perform. Therefore, this voice command may include at least five or ten syllables and take up a duration of one or two seconds, which may correspond to five or ten effective openings and closings of the mouth within one or two seconds. In some other situations, such as coughing and yawning, the user's mouth may have one or two effective openings and closings, so the difference in the number of effective openings and closings within a certain duration can be used as one of the features to distinguish between speaking and other situations. One effective opening and closing of the mouth can correspond to a peak height difference in the effective height difference that characterizes the maximum opening degree of the lips. Therefore, the number of peak height differences in the change curve of the height difference formed between the lips can be used to determine whether the user's mouth shape during speech meets the speaking characteristics.

[0066] Figure 4 A schematic diagram of a height difference variation curve 400 according to an embodiment of the present disclosure is shown. The height difference variation curve 400 may be fitted based on a plurality of discrete height differences formed between the lips in a plurality of face images during speech production by the user. In the height difference variation curve 400, the horizontal axis may be used to represent time t, and the vertical axis may be used to represent the height difference H formed between the lips. The height difference variation curve 400 also shows a predetermined height threshold h.

[0067] In the example, a peak of the height difference H may represent a lip opening and closing. Figure 4 As shown, there are ten double-lip openings and closings within the time range shown in the height difference variation curve 400. However, these ten double-lip openings and closings are not all effective.

[0068] See also Figure 4 , height differences 411, 412, 413, 414, 415, 416 and 417 are peak height differences in the effective height differences that characterize the maximum opening degree of the lips, and these seven peak height differences are all greater than the height threshold h. That is to say, among the ten times of lip opening and closing, only the seven times of lip opening and closing corresponding to height differences 411, 412, 413, 414, 415, 416 and 417 are effective opening and closing.

[0069] Assume that the preset effective opening and closing number threshold is 10, and Figure 4 The number of peak height differences in is 7, which is less than the threshold value. Then it can be determined that the mouth shape of the user during the speech does not meet the speaking characteristics.

[0070] According to an embodiment of the present disclosure, when the number of peak height differences is less than a quantity threshold, it is determined that the user's mouth shape does not conform to the speaking characteristics, which can reduce the interference of coughing, yawning, etc., and to a certain extent avoid erroneously judging that the user is in a state of issuing voice commands in such situations.

[0071] According to some embodiments, when determining whether the mouth shape of the user during speech production meets the speech characteristics, the time interval between each two peak height differences adjacent in time can also be determined in response to determining that the number of peak height differences is greater than or equal to a predetermined number threshold. It can then be determined whether each time interval is less than a predetermined first duration threshold. When it is determined that not every time interval is less than the predetermined first duration threshold, it can be determined that the mouth shape of the user during speech production does not meet the speech characteristics.

[0072] When a user issues a voice command, the mouth generally opens and closes continuously. However, in some other situations, such as yawning, the effective opening and closing may not be continuous, so whether the effective opening and closing is continuous can be used as one of the features to distinguish between speaking and other situations. In some embodiments, the time interval between each two peak height differences adjacent in time can be used to determine whether the effective opening and closing is continuous, thereby determining whether the mouth shape of the user during the speech is consistent with the speaking characteristics.

[0073] Continue to see Figure 4 , assuming that the threshold of the number of effective openings and closings is set to 5, and Figure 4 The number of peak height differences in is 7, which is greater than the number threshold, then the time interval between each two adjacent peak height differences in height differences 411, 412, 413, 414, 415, 416 and 417 can be determined, which are represented as time intervals 421, 422, 423, 424, 425 and 426 respectively.

[0074] Then, it can be further determined whether each of the time intervals 421, 422, 423, 424, 425, and 426 is less than a predetermined first duration threshold. Assuming that the time intervals 421, 424, and 425 are greater than the predetermined first duration threshold, and the time intervals 422, 423, and 426 are less than the first duration threshold, that is, not every time interval is less than the predetermined first duration threshold, it can be determined that the mouth shape of the user during the speech does not meet the speaking characteristics.

[0075] According to the embodiment of the present disclosure, when the effective opening and closing of the user's mouth is discontinuous, it is determined that it does not meet the speaking characteristics, which can reduce interference from situations such as yawning, and further avoid erroneously determining that the user is in a state of issuing voice commands in such situations.

[0076] According to some embodiments, in the process of determining whether the mouth shape of the user during the speech is consistent with the speech characteristics, the difference between each two adjacent time intervals can be determined in response to determining that each time interval is less than a predetermined first duration threshold, and then it can be determined whether there is a difference greater than or equal to a predetermined second duration threshold. When it is determined that there is a difference greater than or equal to the predetermined second duration threshold, it can be determined that the mouth shape of the user during the speech is consistent with the speech characteristics.

[0077] When the user issues a voice command, effective opening and closing is generally not periodic. In some other situations, such as coughing, effective opening and closing may have a certain periodicity, so whether the effective opening and closing has periodicity can also be used as one of the features to distinguish between speaking and other situations. When the difference between each two adjacent time intervals in the above time interval is very small, for example, they are all less than the difference of the predetermined second duration threshold, it can be considered that the effective opening and closing has periodicity. On the contrary, if the difference between each two adjacent time intervals is greater than or equal to the predetermined second duration threshold, it can be considered that the effective opening and closing does not have periodicity, so that it can be determined that the user's mouth shape during the speech is consistent with the speaking characteristics.

[0078] In the example, assuming that time intervals 421, 422, 423, 424, 425 and 426 are all less than a predetermined first duration threshold, it can be further determined whether there is a difference between every two adjacent time intervals that is greater than or equal to a predetermined second duration threshold.

[0079] See also Figure 4 , the differences between time intervals 421 and 422, 422 and 423, and 425 and 426 are small, while the differences between time intervals 423 and 424, and 424 and 425 are large. Assuming that the differences between time intervals 423 and 424, and 424 and 425 are greater than the difference of the predetermined second duration threshold, that is, the difference between every two adjacent time intervals is greater than or equal to the predetermined second duration threshold, and the effective opening and closing does not have a fixed period, it can be determined that the mouth shape of the user during the speech is consistent with the speaking characteristics.

[0080] According to an embodiment of the present disclosure, when the effective opening and closing of the user's mouth shape is not periodic, it is determined that it meets the speaking characteristics, which can reduce interference from situations such as coughing, and further avoid erroneously determining that the user is in a state of issuing voice commands in such situations.

[0081] According to some embodiments, in response to determining that the user's mouth shape during speech conforms to the speech characteristics, the process of determining that the user is in a state of issuing a voice command can be implemented by the following steps. First, in response to determining that the user's mouth shape during speech conforms to the speech characteristics, the angle between the device for acquiring multiple facial images and the user's face relative to the plane where the face is located can be calculated. Then, based on the comparison of the angle with the right angle, it can be determined whether the face is facing the device. In response to determining that the face is facing the device, it can be determined that the user is in a state of issuing a voice command.

[0082] When users communicate with others face to face, they generally face the person they are communicating with. In order to make users feel closer to the feeling and experience of natural communication with people when using voice recognition applications to control vehicle-mounted electromechanical equipment, and reduce the feeling of talking to machines, the device that serves as the carrier of the voice recognition application can be designed to be more humane. When users issue voice commands to it, they usually face the device as if they are communicating naturally with people. However, when users talk to other people in the car or make phone calls, they usually do not face the device. Therefore, it is also possible to determine whether the user is in the state of issuing voice commands by judging whether the user's face is facing the device.

[0083] In an example, the device may include a visual camera for acquiring a facial image of the user during speech and a micro-microphone for acquiring the audio of the user during speech. When the user's face is facing the device, it can be considered that the user's face is also facing the visual camera.

[0084] Figure 5 A schematic diagram of determining whether a face is facing a device according to an embodiment of the present disclosure is shown.

[0085] In an example, the visual camera 510 may capture an image of a conical area with itself as the center. Figure 5 Only a cross section where the visual camera 510 and the user's face are located is shown, so the shooting area of ​​the visual camera 510 is Figure 5 A sector-shaped area is shown in FIG.

[0086] In the example, a line 520 between the center of the lens of the visual camera 510 and a certain position of the user's face, such as the tip of the nose or the center of the eyebrows, can be used as a line between the device and the user's face. An angle θ is formed between the line 520 and the plane 530 where the face is located.

[0087] In the example, when the angle θ is within a predetermined angle range of about 90 degrees, for example, when the angle θ is greater than 80 degrees and less than 100 degrees, it is considered that the user's face is facing the device.

[0088] In the example, the angle θ can be calculated by using the Taylor expansion formula and the tangent theorem. This calculation method belongs to conventional technology and will not be described in detail here.

[0089] According to the embodiments of the present disclosure, by judging whether the user's face is facing the device to determine whether the user is in a state of issuing a voice command, the accuracy of determining the timing of voice recognition can be improved, thereby providing the user with a better experience.

[0090] According to some embodiments, in response to determining that the user's mouth shape during speech is consistent with the speech characteristics, the process of determining that the user is in a state of issuing a voice command can be achieved by the following steps. First, in response to determining that the user's mouth shape during speech is consistent with the speech characteristics, a reference point can be determined in multiple face images, and the reference point can be used to indicate the standard position of the eyes and / or mouth in the face. Then, an identification point can be determined in the multiple face images, and the identification point can be used to indicate the actual position of the eyes and / or mouth in the face. Then, the offset of the identification point in the face image relative to the corresponding reference point can be calculated. Based on the comparison of the offset with a predetermined offset threshold, it can be determined whether the user is looking directly at the device. In response to determining that the user is looking directly at the device, it can be determined that the user is in a state of issuing a voice command.

[0091] When a user is communicating with others face to face, he or she will not only face the person being communicated with, but will also generally look at the other person. When a user issues a voice command to a device, he or she will usually look directly at the device. Therefore, it is also possible to determine whether the user is in a state of issuing a voice command by judging whether the user is looking directly at the device (or the visual camera of the device).

[0092] Figure 6 A schematic diagram of determining whether a user is looking directly at a device according to an embodiment of the present disclosure is shown.

[0093] In the example, see Figure 6 , reference points may be first determined in the face image 600, for example, reference points 610 and 620 may be included to indicate the standard position of the eyes in the face, and reference point 630 may be included to indicate the standard position of the mouth in the face. In some embodiments, the intersection of the length and height of the eyes may be used as reference points 610 and 620, respectively, and the intersection of the length and height of the mouth may be used as reference point 630. Generally speaking, the standard positions of the eyes and the mouth in the face may form an isosceles triangle. Based on such a rule, it is easy to determine the positions of the reference points 610, 620, and 630 in the face.

[0094] by Figure 6Taking the left eye in the face image 600 as an example, an identification point 611 for indicating the actual position of the left eye can be determined in the face image 600. In some embodiments, for example, the pupil center of the left eye can be used as the identification point 611.

[0095] In the example, the offset threshold can be preset to 1 / 20 of the axis length, for example. That is, when the offset of the identification point 611 relative to the reference point 610 on the horizontal axis (i.e., the major axis) is greater than 1 / 20 of the major axis length 612, and / or the offset of the identification point 611 relative to the reference point 610 on the vertical axis (i.e., the minor axis) is greater than 1 / 20 of the minor axis length 613, it can be considered that the offset of the identification point 611 relative to the reference point 610 is greater than the predetermined offset threshold.

[0096] See also Figure 6 , the offset 614 of the identification point 611 relative to the reference point 610 in the horizontal axis direction is approximately 1 / 4 of the long axis length 612, and there is no offset in the vertical axis direction. Therefore, when the offset threshold is preset to 1 / 20 of the axis length, it can be determined that the offset of the identification point 611 relative to the reference point 610 is greater than the predetermined offset threshold, and then it is concluded that the user is not looking directly at the device. Since it is determined that the user is not looking directly at the device, it can be determined that the user is not in a state of issuing a voice command.

[0097] Assuming that the pre-set offset threshold is larger, for example, 1 / 3 of the axis length, it means that the offset of the identification point 611 relative to the reference point 610 is smaller than the offset threshold, and it can be determined that the user is looking directly at the device, and then it can be further determined that the user is in a state of issuing voice commands.

[0098] According to the embodiments of the present disclosure, by judging whether the user is looking directly at the device to determine whether the user is in a state of issuing a voice command, the accuracy of determining the timing of voice recognition can be improved, thereby providing the user with a better experience.

[0099] Figure 7 A schematic diagram of a voice command recognition method according to an embodiment of the present disclosure is shown. The implementation of the voice command recognition method involves a camera device 710 , an audio storage unit 720 , an image analysis unit 730 , a voice recognition unit 740 , and an onboard electromechanical device 750 .

[0100] First, an image segment can be acquired during the user's speech by a camera device 710. The image segment can include multiple face images 702 of the user and audio 701 synchronized with the face images 702. The camera device 710 can include a visual camera and a miniature microphone.

[0101] Then, the camera device 710 can transmit the audio 701 to the audio storage unit 720. The audio 701 can be stored as a recording file by the audio storage unit 720 for recall and recognition. The audio storage unit 720 can periodically overwrite the recording files stored therein to remove useless audio and audio that has been completed by voice recognition. The overwriting period can be determined based on the size of the storage space of the audio storage unit 720, for example, it can be five seconds.

[0102] At the same time, the camera device 710 can also transmit the multiple face images 702 to the image analysis unit 730. The image analysis unit 730 can analyze the mouth shape, face orientation and sight direction in the multiple face images. Based on these analyses, the image analysis unit 730 can determine whether the user is in a state of issuing a voice command during the voice.

[0103] When the image analysis unit 730 determines that the user is in a state of issuing voice commands while uttering voice, the voice recognition unit 740 can call the recording file corresponding to the voice from the audio storage unit 720 and perform voice recognition on the audio in the recording file to convert it into a control instruction 703.

[0104] Finally, the speech recognition unit 740 may send the generated control instruction 703 to the corresponding vehicle-mounted electromechanical device 750 to control it.

[0105] According to another aspect of the present disclosure, a voice command recognition device is also provided.

[0106] Figure 8 A structural block diagram of a voice instruction recognition device 800 according to an embodiment of the present disclosure is shown.

[0107] like Figure 8 As shown, the voice command recognition device 800 includes: an image acquisition module 810, configured to acquire multiple facial images of the user during the period when the user utters a voice; a state judgment module 820, configured to determine whether the user is in a state of issuing a voice command during the period when the user utters a voice based on the analysis of the mouth shape in the multiple facial images, wherein the state determination module 820 includes a speech feature comparison module 821, configured to determine whether the mouth shape of the user during the period when the user utters the voice meets the speech feature; a state determination module 830, configured to determine that the user is in a state of issuing a voice command in response to determining that the mouth shape of the user during the period when the user utters the voice meets the speech feature; an audio acquisition module 840, configured to acquire audio corresponding to the voice; and a voice recognition module 850, configured to recognize the audio to determine the content of the voice command.

[0108] Since the image acquisition module 810, the state judgment module 820, the state determination module 830, the audio acquisition module 840, and the speech recognition module 850 in the speech instruction recognition device 800 can respectively correspond to the following Figure 2 Therefore, the details of each aspect are not repeated here.

[0109] In addition, the voice command recognition device 800 and the modules included therein may further include further submodules, which will be described in conjunction with the following. Fig. 9 Provide detailed explanation.

[0110] According to an embodiment of the present disclosure, by analyzing the mouth shape in the face image during the period when the user utters a voice, it is determined whether the user's mouth shape during this period meets the speaking characteristics, thereby determining whether the user is in a state of issuing a voice command, and the feature analysis of the mouth shape in the face image can be combined to determine the wake-up timing of the voice recognition application. By obtaining audio and performing voice recognition on it only when it is determined that the user is in a state of issuing a voice command, the probability of the voice recognition application being affected by external noise, thereby erroneously triggering the voice recognition application or failing to trigger the voice recognition application can be reduced. At this time, the accuracy of the recognized instruction content can also be improved, and the tasks performed by the on-board electromechanical equipment based on the instruction content can be more in line with the user's expectations, giving the user a better experience.

[0111] Fig. 9 A structural block diagram of a voice instruction recognition device 900 according to another embodiment of the present disclosure is shown.

[0112] like Fig. 9 As shown, the voice command recognition device 900 may include an image acquisition module 910, a state judgment module 920, a state determination module 930, an audio acquisition module 940, and a voice recognition module 950. The image acquisition module 910, the state judgment module 920, the state determination module 930, the audio acquisition module 940, and the voice recognition module 950 may be used together with the following Figure 8 The image acquisition module 810, state judgment module 820, state determination module 830, audio acquisition module 840, and speech recognition module 850 shown correspond to each other, so their details are not repeated here.

[0113] In the example, the speech feature comparison module 921 may include: a height difference determination module 921a, configured to determine, based on multiple face images, a plurality of height differences formed between the lips of a user during speech; an effective height difference determination module 921b, configured to determine whether there is an effective height difference greater than or equal to a predetermined height threshold among the multiple height differences; and a first result determination module 921c, configured to determine that the mouth shape of the user during speech does not conform to the speech feature in response to determining that there is no effective height difference among the multiple height differences.

[0114] In the example, the speech feature comparison module 921 may also include: a peak number determination module 921d, configured to determine the number of peak height differences characterizing the maximum degree of opening of the lips in the effective height differences in response to determining that there are effective height differences among multiple height differences; a peak number comparison module 921e, configured to determine whether the number of peak height differences is greater than or equal to a predetermined number threshold; and a second result determination module 921f, configured to determine that the user's mouth shape during the speech does not meet the speech feature in response to determining that the number of peak height differences is less than a predetermined number threshold.

[0115] In the example, the speech feature comparison module 921 may also include: a peak interval determination module 921g, configured to determine the time interval between each two temporally adjacent peak height differences in response to determining that the number of peak height differences is greater than or equal to a predetermined number threshold; a first duration comparison module 921h, configured to determine whether each time interval is less than a predetermined first duration threshold; and a third result determination module 921i, configured to determine that the user's mouth shape during the speech does not meet the speech feature in response to determining that not every time interval is less than the predetermined first duration threshold.

[0116] In the example, the speech feature comparison module 921 may also include: a time interval difference determination module 921j, configured to determine the difference between each two adjacent time intervals in response to determining that each time interval is less than a predetermined first duration threshold; a second duration comparison module 921k, configured to determine whether there is a difference greater than or equal to a predetermined second duration threshold; and a fourth result determination module 921l, configured to determine whether the user's mouth shape during the speech is consistent with the speech feature in response to determining that there is a difference greater than or equal to the predetermined second duration threshold.

[0117] In the example, the state determination module 930 may include: an angle calculation module 931, configured to calculate the angle between the line between the device for acquiring multiple facial images and the user's face relative to the plane where the face is located in response to determining that the user's mouth shape during speech is consistent with the speaking characteristics; an angle comparison module 932, configured to determine whether the face is facing the device based on a comparison of the angle and a right angle; and a fifth result determination module 933, configured to determine that the user is in a state of issuing a voice command in response to determining that the face is facing the device.

[0118] In the example, the state determination module 930 may include: a reference point determination module 934, configured to determine reference points in multiple face images in response to determining that the user's mouth shape during speech conforms to the speaking characteristics, the reference points are used to indicate the standard position of the eyes and / or mouth in the face; an identification point determination module 935, configured to determine identification points in multiple face images, the identification points are used to indicate the actual position of the eyes and / or mouth in the face; an offset calculation module 936, configured to calculate the offset of the identification point in the face image relative to the corresponding reference point; a line of sight judgment module 937, configured to determine whether the user is looking directly at the device based on a comparison of the offset with a predetermined offset threshold; and a sixth result determination module 938, configured to determine that the user is in a state of issuing a voice command in response to determining that the user is looking directly at the device.

[0119] According to an embodiment of the present disclosure, an electronic device, a readable storage medium, a computer program product and an autonomous driving vehicle are also provided.

[0120] refer to Fig.10 , a block diagram of an electronic device 1000 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0121] like Fig.10As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0122] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009. The input unit 1006 can be any type of device that can input information to the electronic device 1000. The input unit 1006 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 1007 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1008 can include but is not limited to a disk and an optical disk. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0123] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a voice instruction recognition method. For example, in some embodiments, the voice instruction recognition method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the voice instruction recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the voice instruction recognition method in any other appropriate manner (eg, by means of firmware).

[0124] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0126] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0128] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0129] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.

[0131] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for voice command recognition, comprising: Acquire multiple facial images of the user during the period when the user speaks; Based on the analysis of the mouth shapes in the multiple face images, determining whether the user is in a state of issuing a voice command during the period of uttering the voice, wherein the analysis of the mouth shapes in the multiple face images includes determining whether the mouth shape of the user during the period of uttering the voice meets the speaking characteristics; In response to determining that the mouth shape of the user during the utterance of the speech meets the speaking characteristics, determining that the user is in a state of issuing a voice command; Acquire audio corresponding to the voice; and The audio is recognized to determine content of the voice instruction.

2. The method according to claim 1, wherein: The determining whether the mouth shape of the user during the utterance of the speech meets the speaking characteristics comprises: Based on the multiple face images, determining multiple height differences formed between the lips of the user during the utterance of the speech; Determining whether there is a valid height difference among the plurality of height differences that is greater than or equal to a predetermined height threshold; and In response to determining that the valid height difference does not exist in the plurality of height differences, it is determined that the mouth shape of the user during utterance of the speech does not conform to speech characteristics.

3. The method according to claim 2, wherein: The determining whether the mouth shape of the user during the utterance of the speech meets the speaking characteristics further includes: In response to determining that the effective height difference exists in the plurality of height differences, determining the number of peak height differences in the effective height differences that characterize the maximum opening degree of the lips; determining whether the number of peak height differences is greater than or equal to a predetermined number threshold; and In response to determining that the number of the peak height differences is less than the predetermined number threshold, it is determined that the mouth shape of the user during the utterance of the speech does not meet the speaking characteristics.

4. The method according to claim 3, wherein: The determining whether the mouth shape of the user during the utterance of the speech meets the speaking characteristics further includes: In response to determining that the number of the peak height differences is greater than or equal to the predetermined number threshold, determining a time interval between every two peak height differences that are adjacent in time; determining whether each time interval is less than a predetermined first duration threshold; and In response to determining that not every time interval is less than the predetermined first duration threshold, it is determined that the mouth shape of the user during the utterance of the speech does not meet the speaking characteristics.

5. The method according to claim 4, wherein: The determining whether the mouth shape of the user during the utterance of the speech meets the speaking characteristics further includes: In response to determining that each time interval is less than the predetermined first duration threshold, determining a difference between each two adjacent time intervals; determining whether there is a difference greater than or equal to a predetermined second duration threshold; and In response to determining that there is the difference value greater than or equal to the predetermined second duration threshold, it is determined that the mouth shape of the user during the utterance of the speech meets the speaking characteristics.

6. The method according to any one of claims 1 to 5, wherein: In response to determining that the mouth shape of the user during the utterance of the voice meets the speaking feature, determining that the user is in a state of issuing a voice command includes: In response to determining that the mouth shape of the user during the period of uttering the speech meets the speaking feature, calculating the angle between the line between the device for acquiring the plurality of facial images and the face of the user and the plane where the face is located; Based on the comparison between the angle and the right angle, determining whether the face is facing the device; and In response to determining that the human face is facing the device, it is determined that the user is in a state of issuing a voice command.

7. The method according to any one of claims 1 to 6, wherein: In response to determining that the mouth shape of the user during the utterance of the voice meets the speaking feature, determining that the user is in a state of issuing a voice command includes: In response to determining that the mouth shape of the user during the utterance of the speech meets the speaking characteristics, determining reference points in the multiple face images, the reference points being used to indicate standard positions of eyes and / or mouth in the face; Determining identification points in the plurality of facial images, the identification points being used to indicate actual locations of eyes and / or mouths in the faces; Calculating the offset of the recognition point in the face image relative to the corresponding reference point; Determining whether the user is looking directly at the device based on a comparison of the offset with a predetermined offset threshold; and In response to determining that the user is looking directly at the device, it is determined that the user is in a state of issuing a voice command.

8. A voice command recognition device, comprising: An image acquisition module, configured to acquire a plurality of facial images of the user during the period when the user speaks; A state determination module, configured to determine whether the user is in a state of issuing a voice command during the period of uttering the voice based on an analysis of the mouth shapes in the multiple face images, wherein the state determination module includes a speech feature comparison module, configured to determine whether the mouth shape of the user during the period of uttering the voice meets the speech feature; a state determination module, configured to determine that the user is in a state of issuing a voice command in response to determining that the mouth shape of the user during the period of uttering the voice meets the speaking feature; an audio acquisition module, configured to acquire audio corresponding to the voice; and The voice recognition module is configured to recognize the audio to determine the content of the voice instruction.

9. The device according to claim 8, wherein: The speech feature comparison module includes: a height difference determination module, configured to determine, based on the plurality of face images, a plurality of height differences formed between the lips of the user during the utterance of the speech; an effective height difference determination module, configured to determine whether there is an effective height difference greater than or equal to a predetermined height threshold among the plurality of height differences; and The first result determination module is configured to determine that the mouth shape of the user during the utterance of the speech does not meet the speaking characteristics in response to determining that the effective height difference does not exist in the multiple height differences.

10. The device according to claim 9, wherein: The speech feature comparison module also includes: a peak number determination module, configured to determine the number of peak height differences representing the maximum opening degree of the lips among the effective height differences in response to determining that the effective height difference exists among the plurality of height differences; a peak quantity comparison module configured to determine whether the quantity of the peak height differences is greater than or equal to a predetermined quantity threshold; and The second result determination module is configured to determine that the mouth shape of the user during the utterance of the speech does not meet the speaking characteristics in response to determining that the number of the peak height differences is less than the predetermined number threshold.

11. The device according to claim 10, wherein: The speech feature comparison module also includes: a peak interval determination module, configured to determine a time interval between every two peak height differences adjacent in time in response to determining that the number of the peak height differences is greater than or equal to the predetermined number threshold; A first duration comparison module is configured to determine whether each time interval is smaller than a predetermined first duration threshold; and The third result determination module is configured to determine that the mouth shape of the user during the utterance of the speech does not meet the speaking characteristics in response to determining that not every time interval is less than the predetermined first duration threshold.

12. The device according to claim 11, wherein The speech feature comparison module also includes: a time interval difference determination module, configured to determine a difference between every two adjacent time intervals in response to determining that each time interval is less than the predetermined first duration threshold; A second duration comparison module is configured to determine whether there is a difference greater than or equal to a predetermined second duration threshold; and The fourth result determination module is configured to determine that the mouth shape of the user during the utterance of the speech meets the speaking characteristics in response to determining that there is the difference greater than or equal to the predetermined second duration threshold.

13. The device according to any one of claims 8 to 12, wherein: The state determination module comprises: an angle calculation module, configured to calculate an angle between a line between a device for acquiring the plurality of facial images and a face of the user relative to a plane where the face is located, in response to determining that the mouth shape of the user during the period of uttering the speech meets the speech feature; an angle comparison module, configured to determine whether the face is facing the device based on a comparison between the included angle and a right angle; and The fifth result determination module is configured to determine that the user is in a state of issuing a voice command in response to determining that the face is facing the device.

14. The device according to any one of claims 8 to 13, wherein: The state determination module comprises: A reference point determination module, configured to determine reference points in the plurality of face images in response to determining that the mouth shape of the user during the speech is consistent with the speech characteristics, wherein the reference points are used to indicate standard positions of eyes and / or mouth in the face; an identification point determination module, configured to determine identification points in the plurality of face images, the identification points being used to indicate actual positions of eyes and / or mouths in the faces; an offset calculation module, configured to calculate an offset of the identification point in the face image relative to the corresponding reference point; a sight line determination module, configured to determine whether the user is looking directly at the device based on a comparison between the offset and a predetermined offset threshold; and The sixth result determination module is configured to determine that the user is in a state of issuing a voice command in response to determining that the user is looking directly at the device.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 7 when executed by a processor.

18. An autonomous driving vehicle, comprising a controller, wherein the controller is configured to execute the method according to any one of claims 1-7.

Images