Machine vision-based voice input waiting control method and system, and storage medium

Abstract

The application discloses a voice input waiting control method and system based on machine vision and a medium. The method comprises the following steps: in response to a voice recognition activation instruction, acquiring first image information of a target passenger cabin and first voice information of a target passenger; determining whether the voice input is complete according to the first voice information; when it is determined that the voice input is not complete, determining facial expression time sequence data and human body posture time sequence data of the target passenger according to the first image information, and then determining whether the target passenger interrupts speaking according to the facial expression time sequence data and the human body posture time sequence data; when it is determined that the target passenger interrupts speaking, acquiring second voice information of the target passenger within a preset voice input waiting time length, and then synthesizing target voice input information according to the first voice information and the second voice information. The application improves the convenience of voice input and the driving experience of users, and can be widely applied to the technical field of vehicle control.

Description

Technical Field

[0001] The present invention relates to the field of vehicle control technology, and in particular to a method, system and storage medium for voice input waiting control based on machine vision. Background Art

[0002] With the development of intelligent and connected vehicles, vehicle monitoring and control technologies are becoming increasingly intelligent, providing drivers and passengers with an increasingly rich driving experience. Currently, most cars have voice recognition capabilities, which can adjust air conditioning, audio, navigation, and other facilities based on user voice commands. However, when a user is inputting a voice command, they may lose their voice input due to a thought interruption or forgetfulness. In this case, the system cannot correctly recognize the complete voice command and cannot respond appropriately. The user will then be required to re-enter the complete voice command, or may need to repeatedly activate the voice recognition function, affecting the user's driving experience. Summary of the Invention

[0003] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0004] To this end, an object of an embodiment of the present invention is to provide a voice input waiting control method based on machine vision, which improves the convenience of voice input and the user's driving experience.

[0005] Another object of an embodiment of the present invention is to provide a voice input waiting control system based on machine vision.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:

[0007] In a first aspect, an embodiment of the present invention provides a method for controlling voice input waiting based on machine vision, comprising the following steps:

[0008] In response to a voice recognition activation instruction, acquiring first image information of a target passenger compartment and acquiring first voice information of a target passenger;

[0009] Determining whether the voice input is complete according to the first voice information;

[0010] When it is determined that the voice input is incomplete, determining facial expression time series data and human posture time series data of the target occupant based on the first image information, and then determining whether the target occupant has interrupted speaking based on the facial expression time series data and the human posture time series data;

[0011] When it is determined that the target occupant has interrupted speaking, second voice information of the target occupant is obtained within a preset voice input waiting time, and then target voice input information is synthesized based on the first voice information and the second voice information.

[0012] Furthermore, in one embodiment of the present invention, in response to the voice recognition activation instruction, obtaining first image information of the target passenger compartment and obtaining first voice information of the target occupant specifically includes:

[0013] When the voice recognition activation instruction is detected, the vehicle-mounted voice recognition function is activated, and the image acquisition device and the voice acquisition device provided in the target passenger compartment are activated;

[0014] The first image information is acquired through the image acquisition device, and the first voice information is acquired through the voice acquisition device.

[0015] Furthermore, in one embodiment of the present invention, determining whether the voice input is complete according to the first voice information specifically includes:

[0016] Performing voice recognition on the first voice information to obtain first text content;

[0017] Performing semantic analysis on the first text content to obtain a plurality of candidate voice commands, and matching the candidate voice commands according to a preset voice command library;

[0018] If the match is successful, it is determined that the voice input is complete; if the match is unsuccessful, it is determined that the voice input is incomplete.

[0019] Furthermore, in one embodiment of the present invention, determining facial expression time series data and body posture time series data of the target occupant based on the first image information, and then determining whether the target occupant has interrupted speaking based on the facial expression time series data and the body posture time series data, specifically includes:

[0020] Performing face detection on the first image information to obtain a facial expression image, and determining the facial expression time series data based on a plurality of consecutive frames of the facial expression images;

[0021] Performing human body key point detection on the first image information to obtain a human body posture image, and determining the human body posture time series data based on multiple consecutive frames of the human body posture image;

[0022] The facial expression time series data and the human body posture time series data are input into a pre-trained speech interruption recognition model, and whether the target occupant has interrupted his speech is determined based on the recognition result.

[0023] Furthermore, in one embodiment of the present invention, the voice input waiting control method further includes the step of pre-training the speech interruption recognition model, which specifically includes:

[0024] Acquire a plurality of preset speech interruption sample data, wherein the speech interruption sample data includes facial expression time series sample data and human body posture time series sample data;

[0025] Determining label information corresponding to each of the speech interruption sample data, and constructing a training data set based on the speech interruption sample data and the label information;

[0026] Inputting the training data set into a pre-built two-layer bidirectional recurrent neural network for training to obtain the trained speech interruption recognition model;

[0027] The two-layer bidirectional recurrent neural network includes an input layer, a first forward hidden layer, a second forward hidden layer, a first reverse hidden layer, a second reverse hidden layer and an output layer.

[0028] Furthermore, in one embodiment of the present invention, inputting the training data set into a pre-built two-layer bidirectional recurrent neural network for training specifically includes:

[0029] Inputting the speech interruption sample data into the input layer, calculating the facial expression time series sample data through the first forward hidden layer to obtain a first hidden state vector, and calculating the human body posture time series sample data through the second forward hidden layer to obtain a second hidden state vector;

[0030] Inputting the speech interruption sample data into the input layer in reverse order, calculating the third hidden state vector on the reversed facial expression time series sample data through the first reverse hidden layer, and calculating the fourth hidden state vector on the reversed human posture time series sample data through the second reverse hidden layer;

[0031] Reversing the third hidden state vector to obtain a fifth hidden state vector, and reversing the fourth hidden state vector to obtain a sixth hidden state vector;

[0032] concatenating the first hidden state vector and the fifth hidden state vector to obtain a seventh hidden state vector, and concatenating the second hidden state vector and the sixth hidden state vector to obtain an eighth hidden state vector;

[0033] performing weighted averaging processing on the seventh hidden state vector and the eighth hidden state vector according to a preset weight parameter to obtain a ninth hidden state vector, and then inputting the ninth hidden state vector into the output layer to output a speech interruption prediction result;

[0034] Determining a loss value of the two-layer bidirectional recurrent neural network according to the speech interruption prediction result and the label information;

[0035] The model parameters of the two-layer bidirectional recurrent neural network are updated according to the loss value to obtain the trained speech interruption recognition model.

[0036] Furthermore, in one embodiment of the present invention, when it is determined that the target occupant has interrupted speaking, obtaining second voice information of the target occupant within a preset voice input waiting time, and then synthesizing the first voice information and the second voice information to obtain target voice input information, specifically includes:

[0037] When it is determined that the target occupant has interrupted speaking, keeping the vehicle-mounted voice recognition function activated until the voice input waiting time is reached, and acquiring the second voice information within the voice input waiting time;

[0038] Speech recognition and text splicing are performed on the first speech information and the second speech information to obtain the target speech input information.

[0039] In a second aspect, an embodiment of the present invention provides a voice input waiting control system based on machine vision, comprising:

[0040] an information acquisition module, configured to acquire first image information of a target passenger compartment and first voice information of a target passenger in response to a voice recognition activation instruction;

[0041] A first judgment module, configured to judge whether the voice input is complete according to the first voice information;

[0042] a second judgment module, configured to, when determining that the voice input is incomplete, determine facial expression time series data and human posture time series data of the target occupant based on the first image information, and further determine whether the target occupant has interrupted speaking based on the facial expression time series data and the human posture time series data;

[0043] The voice input waiting module is used to obtain the second voice information of the target occupant within a preset voice input waiting time when it is determined that the target occupant has interrupted speaking, and then synthesize the first voice information and the second voice information to obtain the target voice input information.

[0044] In a third aspect, an embodiment of the present invention provides a voice input waiting control device based on machine vision, comprising:

[0045] at least one processor;

[0046] at least one memory for storing at least one program;

[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned machine vision-based voice input waiting control method.

[0048] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing a program executable by a processor, wherein the program executable by the processor is used to execute the above-mentioned machine vision-based voice input waiting control method when executed by the processor.

[0049] The advantages and benefits of the present invention will be described in part in the following description and will become apparent from the following description or learned through practice of the present invention:

[0050] In an embodiment of the present invention, in response to a voice recognition activation instruction, first image information of a target passenger compartment and first voice information of a target occupant are obtained, and whether the voice input is complete is determined based on the first voice information. When it is determined that the voice input is incomplete, facial expression timing data and human posture timing data of the target occupant are determined based on the first image information, and then whether the target occupant's speech is interrupted is determined based on the facial expression timing data and human posture timing data. When it is determined that the target occupant's speech is interrupted, second voice information of the target occupant is obtained within a preset voice input waiting time, and then the target voice input information is synthesized based on the first voice information and the second voice information. The embodiment of the present invention obtains image information of the passenger compartment while activating the voice recognition function to obtain the first voice information of the occupant, and determines whether the voice input is complete based on the first voice information. If the voice input is incomplete, the facial expression timing data and body posture timing data of the occupant are extracted based on the image information, and the facial expression timing data and body posture timing data are combined to determine whether the occupant has interrupted his speech. When it is determined that the occupant has interrupted his speech, the voice recognition function is kept activated within the preset voice input waiting time, so that the occupant can continue to input the second voice information within the voice input waiting time. The complete voice input information can be synthesized based on the first voice information and the second voice information. There is no need to re-enter the complete voice command after the voice input is interrupted, and there is no need to repeatedly wake up the voice recognition function, thereby improving the convenience of voice input and the user's driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0052] Figure 1 A flowchart of a method for controlling voice input waiting based on machine vision provided by an embodiment of the present invention;

[0053] Figure 2 A structural block diagram of a voice input waiting control system based on machine vision provided by an embodiment of the present invention;

[0054] Figure 3 This is a structural block diagram of a machine vision-based voice input waiting control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] In the description of the present invention, "a plurality" means two or more. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly indicating the number of the indicated technical features, or as implicitly indicating the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art.

[0057] Reference Figure 1 The embodiment of the present invention provides a method for controlling voice input waiting based on machine vision, which specifically includes the following steps:

[0058] S101 . In response to a voice recognition activation instruction, obtain first image information of a target passenger compartment and obtain first voice information of a target passenger.

[0059] Specifically, when an occupant wakes up the voice recognition function, the embodiment of the present invention not only obtains the occupant's voice information, but also simultaneously obtains image information of the passenger compartment (authorized in advance by the occupant). The image information is used to determine whether the occupant has interrupted his or her speech when the voice input is incomplete.

[0060] As a further optional embodiment, in response to the voice recognition activation instruction, obtaining first image information of the target passenger compartment and obtaining first voice information of the target occupant specifically includes:

[0061] S1011: When a voice recognition activation command is detected, the vehicle-mounted voice recognition function is activated, and an image acquisition device and a voice acquisition device provided in the target passenger compartment are activated;

[0062] S1012: Acquire first image information through an image acquisition device, and acquire first voice information through a voice acquisition device.

[0063] Specifically, the user can issue a voice recognition activation command through a button, a wake-up word, or a gesture. The vehicle body controller activates the in-vehicle voice recognition function while also activating the image acquisition device and the voice acquisition device. It should be noted that the image acquisition device can be the image acquisition device included in the occupant monitoring system or a separate image acquisition device set for the in-vehicle voice recognition function, and this is not limited in this embodiment of the present invention.

[0064] S102: Determine whether the voice input is complete according to the first voice information.

[0065] Specifically, after collecting a voice message from the passenger, the system determines whether the voice input is complete based on the voice message. If it is complete, it means that the passenger has completed the voice input. If it is incomplete, it is necessary to combine the image information of the passenger compartment to determine whether the passenger has interrupted their speech (interrupted thinking, temporarily forgetting the instructions that need to be issued).

[0066] As an optional implementation, determining whether the voice input is complete according to the first voice information specifically includes:

[0067] S1021. Perform voice recognition on the first voice information to obtain first text content;

[0068] S1022: performing semantic analysis on the first text content to obtain a plurality of candidate voice commands, and matching the candidate voice commands according to a preset voice command library;

[0069] S1023. If the match is successful, determine that the voice input is complete; if the match is unsuccessful, determine that the voice input is incomplete.

[0070] Specifically, inputting voice information into a pre-trained voice recognition model can obtain the corresponding text content, such as "Please help open XX music (music playback platform name)", and "Please help open ——"; performing semantic analysis on the text content can obtain several corresponding candidate voice commands, such as "Open XX music", and "Open"; then matching the candidate voice commands according to a preset voice command library, which pre-stores different operation actions and corresponding operation objects. When the candidate voice command matches the corresponding operation action and operation object in the voice command library, it indicates a successful match.

[0071] The voice command library includes operations such as "open", "close", "start", "play", "adjust up", "adjust down", "pause", etc. Each operation action is configured with a corresponding operation object. For example, the operation objects corresponding to the operation action "open" include "air conditioning", "navigation", "windows", "XX music", etc. For the candidate voice command "open XX music", the operation action "open" is matched first, and then the operation object "XX music" corresponding to the operation action "open" is matched, indicating that the candidate voice command "open XX music" is matched successfully. For the candidate voice command "open", if only the operation action can be matched, the match is unsuccessful.

[0072] It should be noted that the speech recognition model of the embodiment of the present invention can adopt the existing neural network model in the prior art, and the embodiment of the present invention will not be described in detail here.

[0073] S103. When it is determined that the voice input is incomplete, determine the facial expression time series data and the body posture time series data of the target occupant based on the first image information, and then determine whether the target occupant has interrupted speaking based on the facial expression time series data and the body posture time series data.

[0074] Specifically, when the voice input is incomplete, it may be due to interruption of thinking, temporary forgetting of the instructions to be issued, or interruption by other things, causing the occupant to interrupt his speech (i.e., the voice input needs to be continued later), or it may be that the occupant actively gives up the voice input (i.e., the speaking is stopped and no more voice input is made). The embodiment of the present invention determines whether the occupant has interrupted his speech based on the occupant's facial expression time series data and body posture time series data, so as to facilitate the control execution of subsequent voice input waiting.

[0075] As a further optional embodiment, determining facial expression time series data and human posture time series data of the target occupant based on the first image information, and then determining whether the target occupant has interrupted speaking based on the facial expression time series data and human posture time series data, specifically includes:

[0076] S1031, performing face detection on the first image information to obtain a facial expression image, and determining facial expression time series data based on multiple consecutive frames of facial expression images;

[0077] S1032, performing human body key point detection on the first image information to obtain a human body posture image, and determining human body posture time series data based on multiple consecutive frames of human body posture images;

[0078] S1033. Input the facial expression time series data and the human body posture time series data into a pre-trained speech interruption recognition model, and determine whether the target occupant has interrupted his speech based on the recognition result.

[0079] Specifically, face detection is performed on the first image information using a pre-trained face detection model, and a facial expression image is obtained based on the recognized face area. Multiple frames of facial expression images can form facial expression time series data; human key point detection is performed on the first image information using a pre-trained human key point detection model, and a human posture image can be obtained based on the positions of recognized key points such as the head, upper torso, and hands. Multiple frames of human posture images can form human posture time series data; the facial expression time series data and the human posture time series data are input into a pre-trained speech interruption recognition model to determine whether the target occupant has interrupted his speech.

[0080] It should be noted that the face detection model and the human key point detection model of the embodiment of the present invention can adopt the existing neural network model in the prior art, and the embodiment of the present invention will not be described in detail here. The speech interruption recognition model of the embodiment of the present invention is obtained by collecting sample data formed by historical voice interaction data and using a neural network to learn the association between the user's language expression intention and his facial expression and body posture. It can be recognized that when the user's thinking is interrupted, he temporarily forgets the instructions he needs to issue, or his speech is interrupted by other things, he will generally have expressions / actions such as keeping his mouth open, moving his eyeballs, looking up, holding his forehead with his hand, and scratching the back of his head. The embodiment of the present invention uses a neural network to learn the association between these expressions and actions and whether the user has interrupted his speech, thereby training a speech interruption recognition model. It is more advantageous for children with weaker language logic but richer expressions and behaviors.

[0081] As an optional embodiment, the voice input waiting control method further includes the step of pre-training a speech interruption recognition model, which specifically includes:

[0082] S201, obtaining a plurality of preset speech interruption sample data, where the speech interruption sample data includes facial expression time series sample data and human body posture time series sample data;

[0083] S202: Determine label information corresponding to each speech interruption sample data, and construct a training data set based on the speech interruption sample data and the label information;

[0084] S203, inputting the training data set into a pre-built two-layer bidirectional recurrent neural network for training to obtain a trained speech interruption recognition model;

[0085] The two-layer bidirectional recurrent neural network includes an input layer, a first forward hidden layer, a second forward hidden layer, a first reverse hidden layer, a second reverse hidden layer and an output layer.

[0086] Specifically, when constructing the training dataset, the team obtains time-series facial expression and body posture sample data from historical occupants of the test vehicle when they used the voice interaction function. The acquisition method for these data is similar to the aforementioned facial expression and body posture data. Furthermore, based on actual conditions (i.e., whether the historical occupant subsequently repeated input) and manual annotation, the team determines the label information corresponding to each speech interruption sample data. This label information indicates whether the historical occupant's speech interruption or speech termination occurred. The training dataset is generated based on the speech interruption sample data and the corresponding label information.

[0087] The speech interruption recognition model of the embodiment of the present invention is obtained by training a two-layer bidirectional recurrent neural network, which includes two hidden layers for feature extraction of facial expression time series data and human posture time series data, respectively, and the two hidden layers share an input layer and an output layer; each hidden layer is divided into a positive hidden layer and a negative hidden layer, which are respectively used to perform positive hidden state calculation and reverse hidden state calculation on the facial expression time series data / human posture time series data; after splicing the hidden state vectors obtained by calculating the bidirectional hidden states within each hidden layer, the splicing vectors output by each hidden layer are weighted averaged based on a preset weighting parameter to obtain the final hidden state vector and output the corresponding speech interruption recognition result through the output layer.

[0088] As an optional implementation, the training data set is input into a pre-built two-layer bidirectional recurrent neural network for training, which specifically includes:

[0089] S2031, inputting speech interruption sample data into the input layer, calculating the facial expression time series sample data through the first forward hidden layer to obtain a first hidden state vector, and calculating the human body posture time series sample data through the second forward hidden layer to obtain a second hidden state vector;

[0090] S2032, inputting the speech interruption sample data into the input layer in reverse order, calculating the reversed facial expression time series sample data through the first reverse hidden layer to obtain a third hidden state vector, and calculating the reversed human posture time series sample data through the second reverse hidden layer to obtain a fourth hidden state vector;

[0091] S2033. Reverse the third hidden state vector to obtain a fifth hidden state vector, and reverse the fourth hidden state vector to obtain a sixth hidden state vector.

[0092] S2034: Concatenate the first hidden state vector and the fifth hidden state vector to obtain a seventh hidden state vector, and concatenate the second hidden state vector and the sixth hidden state vector to obtain an eighth hidden state vector;

[0093] S2035: performing weighted averaging on the seventh hidden state vector and the eighth hidden state vector according to a preset weight parameter to obtain a ninth hidden state vector, and then inputting the ninth hidden state vector into an output layer to obtain a speech interruption prediction result;

[0094] S2036, determining a loss value of a two-layer bidirectional recurrent neural network based on the speech interruption prediction result and the label information;

[0095] S2037. Update the model parameters of the two-layer bidirectional recurrent neural network according to the loss value to obtain a trained speech interruption recognition model.

[0096] Specifically, embodiments of the present invention can assign weights to facial expression time series data and human posture time series data based on an attention mechanism. This weight assignment can be used to determine the impact of different data on speech interruption recognition results, thereby improving the accuracy of speech interruption recognition. The specific weight values ​​can be pre-set. After inputting data from a training dataset with weight parameters into an initialized two-layer bidirectional recurrent neural network, the model outputs a recognition result, namely a speech interruption prediction result. The accuracy of the model prediction can be evaluated based on the speech interruption prediction result and the aforementioned label information, thereby updating the model parameters. For a speech interruption recognition model, the accuracy of the model prediction result can be measured using a loss function. The loss function is defined on a single training data point and is used to measure the prediction error of a training data point. Specifically, the loss value of a training data point is determined by combining the label of the training data point with the model's prediction result for that training data point. In actual training, a training data set contains a large amount of training data. Therefore, a cost function is generally used to measure the overall error of the training data set. The cost function is defined on the entire training data set and is used to calculate the average prediction error of all training data points, which can better measure the prediction effect of the model. For general machine learning models, the cost function mentioned above, plus a regularization term that measures the complexity of the model, can be used as the objective function for training. Based on this objective function, the loss value of the entire training data set can be calculated. There are many types of commonly used loss functions, such as 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross entropy loss function, etc., which can all be used as loss functions for machine learning models, which will not be elaborated one by one here. In an embodiment of the present invention, any one of the loss functions can be selected to determine the loss value for training. Based on the loss value of training, the backpropagation algorithm is used to update the parameters of the model, and a trained speech interruption recognition model can be obtained after several rounds of iteration. The specific number of iterations can be pre-set, or the training is considered to be completed when the test set meets the accuracy requirements.

[0097] S104: When it is determined that the target occupant has interrupted his speech, obtain the second voice information of the target occupant within a preset voice input waiting time, and then synthesize the first voice information and the second voice information to obtain the target voice input information.

[0098] Specifically, when it is determined that the target occupant has interrupted his speech, the on-board voice recognition function is controlled to temporarily not make a judgment on invalid voice commands, and the on-board voice recognition function is kept in an activated state until a preset voice input waiting time is reached (such as 5s, which can be set manually). During the period when the on-board voice recognition function is kept in an activated state, the second voice information of the occupant can continue to be obtained, and then the first voice information and the second voice information can be synthesized to obtain the target voice input information.

[0099] As a further optional implementation, when it is determined that the target occupant has interrupted speaking, obtaining a second voice message of the target occupant within a preset voice input waiting time, and then synthesizing the first voice message and the second voice message to obtain the target voice input message, which specifically includes:

[0100] S1041: When it is determined that the target occupant has interrupted speaking, keep the vehicle-mounted voice recognition function activated until the voice input waiting time expires, and obtain second voice information within the voice input waiting time.

[0101] S1042: Perform voice recognition and text splicing on the first voice information and the second voice information to obtain target voice input information.

[0102] In some optional embodiments, when it is determined that the occupant has stopped speaking, or when the occupant's second voice information has not been obtained when the preset voice input waiting time is reached, the vehicle-mounted voice recognition function is controlled to switch to a dormant state.

[0103] The above describes the method steps of an embodiment of the present invention. It can be appreciated that, in the embodiment of the present invention, while activating the voice recognition function to obtain the first voice information of the occupant, the image information of the passenger compartment is obtained, and whether the voice input is complete is determined based on the first voice information. If the voice input is incomplete, the facial expression time series data and human posture time series data of the occupant are extracted based on the image information, and the facial expression time series data and human posture time series data are combined to determine whether the occupant has interrupted his speech. When it is determined that the occupant has interrupted his speech, the voice recognition function is kept activated within the preset voice input waiting time, so that the occupant can continue to input the second voice information within the voice input waiting time. The complete voice input information can be synthesized based on the first voice information and the second voice information, eliminating the need to re-enter the complete voice command after the voice input is interrupted, and eliminating the need to repeatedly wake up the voice recognition function, thereby improving the convenience of voice input and the user's driving experience.

[0104] Reference Figure 2 , an embodiment of the present invention provides a voice input waiting control system based on machine vision, comprising:

[0105] an information acquisition module, configured to acquire first image information of a target passenger compartment and first voice information of a target passenger in response to a voice recognition activation instruction;

[0106] A first judgment module, configured to judge whether the voice input is complete according to the first voice information;

[0107] a second judgment module, configured to, when determining that the voice input is incomplete, determine facial expression time series data and human posture time series data of the target occupant based on the first image information, and further determine whether the target occupant has interrupted his speech based on the facial expression time series data and human posture time series data;

[0108] The voice input waiting module is used to obtain the second voice information of the target occupant within a preset voice input waiting time when it is determined that the target occupant has interrupted speaking, and then synthesize the target voice input information based on the first voice information and the second voice information.

[0109] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0110] Reference Figure 3 , an embodiment of the present invention provides a voice input waiting control device based on machine vision, comprising:

[0111] at least one processor;

[0112] at least one memory for storing at least one program;

[0113] When the at least one program is executed by the at least one processor, the at least one processor implements the machine vision-based voice input waiting control method.

[0114] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] An embodiment of the present invention further provides a computer-readable storage medium storing a program executable by a processor. When the program is executed by the processor, it is used to execute the above-mentioned method for controlling voice input waiting based on machine vision.

[0116] A computer-readable storage medium according to an embodiment of the present invention can execute a machine vision-based voice input waiting control method provided by an embodiment of the method of the present invention, can execute any combination of implementation steps of the embodiment of the method, and has the corresponding functions and beneficial effects of the method.

[0117] The embodiment of the present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 1 The method shown.

[0118] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0119] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present invention set forth in the claims using ordinary skills without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0120] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0121] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0122] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable media on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0123] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0124] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0125] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0126] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A voice input waiting control method based on machine vision, characterized in that: The following steps are involved: In response to a voice recognition activation instruction, acquiring first image information of a target passenger compartment and acquiring first voice information of a target passenger; Determining whether the voice input is complete according to the first voice information; When it is determined that the voice input is incomplete, determining facial expression time series data and human posture time series data of the target occupant based on the first image information, and then determining whether the target occupant has interrupted speaking based on the facial expression time series data and the human posture time series data; When it is determined that the target occupant has interrupted speaking, second voice information of the target occupant is obtained within a preset voice input waiting time, and then target voice input information is synthesized based on the first voice information and the second voice information.

2. The method for controlling voice input waiting based on machine vision according to claim 1, characterized in that: The method of acquiring first image information of a target passenger compartment and first voice information of a target passenger in response to a voice recognition activation instruction specifically includes: When the voice recognition activation instruction is detected, the vehicle-mounted voice recognition function is activated, and the image acquisition device and the voice acquisition device provided in the target passenger compartment are activated; The first image information is acquired through the image acquisition device, and the first voice information is acquired through the voice acquisition device.

3. The method for controlling voice input waiting based on machine vision according to claim 1, characterized in that: The determining whether the voice input is complete according to the first voice information specifically includes: Performing voice recognition on the first voice information to obtain first text content; Performing semantic analysis on the first text content to obtain a plurality of candidate voice commands, and matching the candidate voice commands according to a preset voice command library; If the match is successful, it is determined that the voice input is complete; if the match is unsuccessful, it is determined that the voice input is incomplete.

4. The method for controlling voice input waiting based on machine vision according to claim 1, characterized in that: The determining of facial expression time series data and human posture time series data of the target occupant based on the first image information, and further determining whether the target occupant has interrupted speaking based on the facial expression time series data and the human posture time series data, specifically includes: Performing face detection on the first image information to obtain a facial expression image, and determining the facial expression time series data based on a plurality of consecutive frames of the facial expression images; Performing human body key point detection on the first image information to obtain a human body posture image, and determining the human body posture time series data based on multiple consecutive frames of the human body posture image; The facial expression time series data and the human body posture time series data are input into a pre-trained speech interruption recognition model, and whether the target occupant has interrupted his speech is determined based on the recognition result.

5. The method for controlling voice input waiting based on machine vision according to claim 4, characterized in that: The voice input waiting control method further includes the step of pre-training the speech interruption recognition model, which specifically includes: Acquire a plurality of preset speech interruption sample data, wherein the speech interruption sample data includes facial expression time series sample data and human body posture time series sample data; Determining label information corresponding to each of the speech interruption sample data, and constructing a training data set based on the speech interruption sample data and the label information; Inputting the training data set into a pre-built two-layer bidirectional recurrent neural network for training to obtain the trained speech interruption recognition model; The two-layer bidirectional recurrent neural network includes an input layer, a first forward hidden layer, a second forward hidden layer, a first reverse hidden layer, a second reverse hidden layer and an output layer.

6. The method for controlling voice input waiting based on machine vision according to claim 5, characterized in that: The inputting of the training data set into a pre-built two-layer bidirectional recurrent neural network for training specifically includes: Inputting the speech interruption sample data into the input layer, calculating the facial expression time series sample data through the first forward hidden layer to obtain a first hidden state vector, and calculating the human body posture time series sample data through the second forward hidden layer to obtain a second hidden state vector; Inputting the speech interruption sample data into the input layer in reverse order, calculating the third hidden state vector on the reversed facial expression time series sample data through the first reverse hidden layer, and calculating the fourth hidden state vector on the reversed human posture time series sample data through the second reverse hidden layer; Reversing the third hidden state vector to obtain a fifth hidden state vector, and reversing the fourth hidden state vector to obtain a sixth hidden state vector; concatenating the first hidden state vector and the fifth hidden state vector to obtain a seventh hidden state vector, and concatenating the second hidden state vector and the sixth hidden state vector to obtain an eighth hidden state vector; performing weighted averaging processing on the seventh hidden state vector and the eighth hidden state vector according to a preset weight parameter to obtain a ninth hidden state vector, and then inputting the ninth hidden state vector into the output layer to output a speech interruption prediction result; Determining a loss value of the two-layer bidirectional recurrent neural network according to the speech interruption prediction result and the label information; The model parameters of the two-layer bidirectional recurrent neural network are updated according to the loss value to obtain the trained speech interruption recognition model.

7. A method for controlling voice input waiting based on machine vision according to any one of claims 1 to 6, characterized in that: When it is determined that the target occupant has interrupted speaking, obtaining second voice information of the target occupant within a preset voice input waiting time, and then synthesizing the first voice information and the second voice information to obtain target voice input information, which specifically includes: When it is determined that the target occupant has interrupted speaking, keeping the vehicle-mounted voice recognition function activated until the voice input waiting time is reached, and acquiring the second voice information within the voice input waiting time; Speech recognition and text splicing are performed on the first speech information and the second speech information to obtain the target speech input information.

8. A voice input waiting control system based on machine vision, characterized in that: include: an information acquisition module, configured to acquire first image information of a target passenger compartment and first voice information of a target passenger in response to a voice recognition activation instruction; A first judgment module, configured to judge whether the voice input is complete according to the first voice information; a second judgment module, configured to, when determining that the voice input is incomplete, determine facial expression time series data and human posture time series data of the target occupant based on the first image information, and further determine whether the target occupant has interrupted speaking based on the facial expression time series data and the human posture time series data; The voice input waiting module is used to obtain the second voice information of the target occupant within a preset voice input waiting time when it is determined that the target occupant has interrupted speaking, and then synthesize the first voice information and the second voice information to obtain the target voice input information.

9. A voice input waiting control device based on machine vision, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the machine vision-based voice input waiting control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to execute the machine vision-based voice input waiting control method according to any one of claims 1 to 7 when executed by the processor.

Images