Voice input waiting control method and system based on machine vision and storage medium
By combining machine vision technology, we can judge the integrity of occupant voice input and speech interruption, complete voice commands that do not need to be re-entered in the car voice recognition system are achieved, improving the user experience.
Patent Information
- Application Number
- CN202411136371.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-19
AI Technical Summary
The existing car voice recognition system cannot correctly recognize the user's voice input when the user's voice input is incomplete, resulting in the user's need to re-enter, affecting the driving experience.
Using a voice input waiting control method based on machine vision, the voice input is determined by obtaining the image information and voice information of the passenger compartment, and the voice input is determined by using facial expressions and human posture timing data to determine whether the occupant speaks interrupted. If it is interrupted, it will be within the preset time. Continue to obtain voice information to synthesize complete input.
It improves the convenience of voice input and user driving experience, avoiding the need for re-entering and repeatedly wake-up voice recognition functions.
Smart Images

Figure CN119091881A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of vehicle control technology, and in particular to a method, system and storage medium for voice input waiting control based on machine vision. Background Art
[0002] With the development of intelligent networking of automobiles, vehicle monitoring and control technologies are becoming more and more intelligent, bringing more and more rich driving experience to drivers and passengers. At present, most cars have voice recognition functions, which can adjust air conditioning, audio, navigation and other facilities according to the voice commands issued by the user. However, when the user is inputting voice, the voice command input may be incomplete due to interruption of thinking or temporary forgetfulness. At this time, the system cannot correctly recognize the complete voice command and cannot make a correct response. The user will need to re-enter the complete voice command later, and may also need to repeatedly wake up the voice recognition function, affecting the user's driving experience. Summary of the invention
[0003] The purpose of the present invention is to solve one of the technical problems existing in the prior art to at least a certain extent.
[0004] To this end, an object of an embodiment of the present invention is to provide a voice input waiting control method based on machine vision, which improves the convenience of voice input and the user's driving experience.
[0005] Another object of an embodiment of the present invention is to provide a voice input waiting control system based on machine vision.
[0006] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present invention include:
[0007] In a first aspect, an embodiment of the present invention provides a method for controlling voice input waiting based on machine vision, comprising the following steps:
[0008] In response to the voice recognition activation instruction, acquiring first image information of a target passenger compartment and acquiring first voice information of a target passenger;
[0009] Determining whether the voice input is complete according to the first voice information;
[0010] When it is determined that the voice input is incomplete, determining the facial expression time series data and the human posture time series data of the target occupant according to the first image information, and then judging whether the target occupant has interrupted speaking according to the facial expression time series data and the human posture time series data;
[0011] When it is determined that the target occupant has interrupted speaking, second voice information of the target occupant is obtained within a preset voice input waiting time, and then the target voice input information is synthesized according to the first voice information and the second voice information.
[0012] Further, in one embodiment of the present invention, in response to the voice recognition activation instruction, acquiring first image information of the target passenger compartment and acquiring first voice information of the target occupant specifically includes:
[0013] When the voice recognition activation instruction is detected, the vehicle-mounted voice recognition function is activated, and the image acquisition device and the voice acquisition device arranged in the target passenger compartment are activated;
[0014] The first image information is acquired through the image acquisition device, and the first voice information is acquired through the voice acquisition device.
[0015] Further, in an embodiment of the present invention, the determining whether the voice input is complete according to the first voice information specifically includes:
[0016] Performing voice recognition on the first voice information to obtain first text content;
[0017] Performing semantic analysis on the first text content to obtain a plurality of candidate voice instructions, and matching the candidate voice instructions according to a preset voice instruction library;
[0018] If the match is successful, it is determined that the voice input is complete; if the match is unsuccessful, it is determined that the voice input is incomplete.
[0019] Further, in one embodiment of the present invention, determining the facial expression time series data and the human posture time series data of the target occupant according to the first image information, and then judging whether the target occupant has interrupted speaking according to the facial expression time series data and the human posture time series data, specifically includes:
[0020] Performing face detection on the first image information to obtain a facial expression image, and determining the facial expression time series data according to a plurality of consecutive frames of the facial expression images;
[0021] Performing human body key point detection on the first image information to obtain a human body posture image, and determining the human body posture time series data according to a plurality of consecutive frames of the human body posture images;
[0022] The facial expression time series data and the human body posture time series data are input into a pre-trained speech interruption recognition model, and whether the target occupant interrupts speech is determined according to the recognition result.
[0023] Furthermore, in one embodiment of the present invention, the voice input waiting control method further includes the step of pre-training the speech interruption recognition model, which specifically includes:
[0024] Acquire a plurality of preset speech interruption sample data, wherein the speech interruption sample data includes facial expression time series sample data and human body posture time series sample data;
[0025] Determining label information corresponding to each of the speech interruption sample data, and constructing a training data set according to the speech interruption sample data and the label information;
[0026] Inputting the training data set into a pre-built two-layer bidirectional recurrent neural network for training to obtain the trained speech interruption recognition model;
[0027] The two-layer bidirectional recurrent neural network includes an input layer, a first forward hidden layer, a second forward hidden layer, a first reverse hidden layer, a second reverse hidden layer and an output layer.
[0028] Further, in one embodiment of the present invention, the inputting the training data set into a pre-built two-layer bidirectional recurrent neural network for training specifically includes:
[0029] Inputting the speech interruption sample data into the input layer, calculating the facial expression time series sample data through the first forward hidden layer to obtain a first hidden state vector, and calculating the human body posture time series sample data through the second forward hidden layer to obtain a second hidden state vector;
[0030] Inputting the speech interruption sample data into the input layer in reverse order, calculating the third hidden state vector on the reversed facial expression time series sample data through the first reverse hidden layer, and calculating the fourth hidden state vector on the reversed human body posture time series sample data through the second reverse hidden layer;
[0031] Reverse processing is performed on the third hidden state vector to obtain a fifth hidden state vector, and reverse processing is performed on the fourth hidden state vector to obtain a sixth hidden state vector;
[0032] Concatenate the first hidden state vector and the fifth hidden state vector to obtain a seventh hidden state vector, and concatenate the second hidden state vector and the sixth hidden state vector to obtain an eighth hidden state vector;
[0033] Performing weighted averaging processing on the seventh hidden state vector and the eighth hidden state vector according to a preset weight parameter to obtain a ninth hidden state vector, and then inputting the ninth hidden state vector into the output layer to obtain a speech interruption prediction result;
[0034] Determining a loss value of the two-layer bidirectional recurrent neural network according to the speech interruption prediction result and the label information;
[0035] The model parameters of the two-layer bidirectional recurrent neural network are updated according to the loss value to obtain the trained speech interruption recognition model.
[0036] Further, in one embodiment of the present invention, when it is determined that the target occupant has interrupted speaking, obtaining second voice information of the target occupant within a preset voice input waiting time, and then synthesizing the first voice information and the second voice information to obtain target voice input information, specifically includes:
[0037] When it is determined that the target occupant has interrupted speaking, keeping the vehicle-mounted voice recognition function in an activated state until the voice input waiting time is reached, and acquiring the second voice information within the voice input waiting time;
[0038] Perform voice recognition and text concatenation on the first voice information and the second voice information to obtain the target voice input information.
[0039] In a second aspect, an embodiment of the present invention provides a voice input waiting control system based on machine vision, comprising:
[0040] an information acquisition module, for acquiring first image information of a target passenger compartment and first voice information of a target passenger in response to a voice recognition activation instruction;
[0041] A first judgment module, used for judging whether the voice input is complete according to the first voice information;
[0042] A second judgment module is used to determine the facial expression time series data and the human posture time series data of the target occupant according to the first image information when it is determined that the voice input is incomplete, and then determine whether the target occupant is interrupted in speaking according to the facial expression time series data and the human posture time series data;
[0043] The voice input waiting module is used to obtain the second voice information of the target occupant within a preset voice input waiting time when it is determined that the target occupant has interrupted speaking, and then synthesize the first voice information and the second voice information to obtain the target voice input information.
[0044] In a third aspect, an embodiment of the present invention provides a voice input waiting control device based on machine vision, comprising:
[0045] at least one processor;
[0046] at least one memory for storing at least one program;
[0047] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned machine vision-based voice input waiting control method.
[0048] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to execute the above-mentioned machine vision-based voice input waiting control method when executed by the processor.
[0049] The advantages and beneficial effects of the present invention will be partly given in the following description, partly become apparent from the following description, or be understood through the practice of the present invention:
[0050] In response to a voice recognition activation instruction, an embodiment of the present invention obtains first image information of a target passenger compartment and first voice information of a target occupant, determines whether the voice input is complete based on the first voice information, and when it is determined that the voice input is incomplete, determines facial expression timing data and human posture timing data of the target occupant based on the first image information, and then determines whether the target occupant has interrupted his speech based on the facial expression timing data and the human posture timing data, and when it is determined that the target occupant has interrupted his speech, obtains second voice information of the target occupant within a preset voice input waiting time, and then synthesizes the first voice information and the second voice information to obtain the target voice input information. The embodiment of the present invention obtains image information of the passenger compartment while activating the voice recognition function to obtain the first voice information of the occupant, and determines whether the voice input is complete based on the first voice information. If the voice input is incomplete, the facial expression timing data and the body posture timing data of the occupant are extracted based on the image information, and the facial expression timing data and the body posture timing data are combined to determine whether the occupant has interrupted his speech. When it is determined that the occupant has interrupted his speech, the voice recognition function is kept activated within a preset voice input waiting time, so that the occupant can continue to input the second voice information within the voice input waiting time, and the complete voice input information can be synthesized based on the first voice information and the second voice information. There is no need to re-enter the complete voice command after the voice input is interrupted, and there is no need to repeatedly wake up the voice recognition function, thereby improving the convenience of voice input and the user's driving experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] In order to more clearly illustrate the technical solution in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solution of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0052] Figure 1 A flowchart of a method for controlling voice input waiting based on machine vision provided by an embodiment of the present invention;
[0053] Figure 2 A structural block diagram of a voice input waiting control system based on machine vision provided by an embodiment of the present invention;
[0054] Figure 3 A structural block diagram of a machine vision-based voice input waiting control device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0055] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0056] In the description of the present invention, the meaning of "a plurality" is two or more than two. If there is a description of "a first" or "a second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used in this document have the same meaning as those commonly understood by those skilled in the art.
[0057] Reference Figure 1 The embodiment of the present invention provides a method for controlling voice input waiting based on machine vision, which specifically includes the following steps:
[0058] S101. In response to a voice recognition activation instruction, first image information of a target passenger compartment is acquired, and first voice information of a target passenger is acquired.
[0059] Specifically, when an occupant wakes up the voice recognition function, the embodiment of the present invention not only obtains the occupant's voice information, but also obtains image information of the passenger compartment (authorized in advance by the occupant). The image information is used to determine whether the occupant has interrupted his speech when the voice input is incomplete.
[0060] As an optional implementation, in response to the voice recognition activation instruction, obtaining first image information of the target passenger compartment and obtaining first voice information of the target passenger may specifically include:
[0061] S1011, when a voice recognition activation command is detected, the vehicle-mounted voice recognition function is started, and an image acquisition device and a voice acquisition device set in the target passenger compartment are started;
[0062] S1012: Acquire first image information through an image acquisition device, and acquire first voice information through a voice acquisition device.
[0063] Specifically, the user can issue a voice recognition activation command through a button, a wake-up word, a gesture, etc., and the body controller starts the in-vehicle voice recognition function while also starting the image acquisition device and the voice acquisition device. It should be noted that the image acquisition device can be an image acquisition device provided by the occupant monitoring system, or an image acquisition device separately provided for the in-vehicle voice recognition function, and the embodiment of the present invention is not limited here.
[0064] S102: Determine whether the voice input is complete according to the first voice information.
[0065] Specifically, after a voice message from the occupant is collected, it is determined whether the voice input is complete based on the voice message. If it is complete, it means that the occupant has completed the voice input. If it is incomplete, it is necessary to combine the image information of the passenger compartment to determine whether the occupant has interrupted his speech (interrupted thinking, temporarily forgetting the instructions that need to be issued).
[0066] Further as an optional implementation manner, judging whether the voice input is complete according to the first voice information specifically includes:
[0067] S1021. Perform voice recognition on the first voice information to obtain first text content;
[0068] S1022, performing semantic analysis on the first text content to obtain a plurality of candidate voice instructions, and matching the candidate voice instructions according to a preset voice instruction library;
[0069] S1023. If the match is successful, determine that the voice input is complete; if the match is unsuccessful, determine that the voice input is incomplete.
[0070] Specifically, inputting voice information into a pre-trained voice recognition model can obtain corresponding text content, such as "Please help open XX music (music playback platform name)", and "Please help open ——"; performing semantic analysis on the text content can obtain several corresponding candidate voice commands, such as "Open XX music", and "Open"; then matching the candidate voice commands according to a preset voice command library, which pre-stores different operation actions and corresponding operation objects. When the candidate voice command matches the corresponding operation action and operation object in the voice command library, it indicates a successful match.
[0071] The voice command library has operation actions such as "open", "close", "start", "play", "adjust up", "adjust down", "pause", etc. Each operation action is configured with a corresponding operation object. For example, the operation objects corresponding to the "open" operation action are "air conditioning", "navigation", "window", "XX music", etc. For the candidate voice command "open XX music", the operation action "open" is matched first, and then the operation object "XX music" corresponding to the operation action "open" is matched, indicating that the candidate voice command "open XX music" is matched successfully. For the candidate voice command "open", only the operation action can be matched, and the match is unsuccessful.
[0072] It should be noted that the speech recognition model of the embodiment of the present invention may adopt an existing neural network model in the prior art, and the embodiment of the present invention will not be described in detail here.
[0073] S103, when it is determined that the voice input is incomplete, determine the facial expression timing data and the body posture timing data of the target occupant according to the first image information, and then judge whether the target occupant interrupts speaking according to the facial expression timing data and the body posture timing data.
[0074] Specifically, when the voice input is incomplete, it may be due to interruption of thinking, temporary forgetting of the command to be issued, or interruption by other things, causing the occupant to interrupt speaking (i.e., the voice input needs to be continued later), or it may be that the occupant actively gives up the voice input (i.e., the speaking is stopped and no more voice input is done). The embodiment of the present invention determines whether the occupant has interrupted speaking based on the occupant's facial expression timing data and body posture timing data, so as to facilitate the control execution of subsequent voice input waiting.
[0075] As an optional implementation, determining facial expression time series data and human posture time series data of the target occupant according to the first image information, and then judging whether the target occupant interrupts his speech according to the facial expression time series data and human posture time series data, specifically includes:
[0076] S1031, performing face detection on the first image information to obtain a facial expression image, and determining facial expression time series data according to a plurality of consecutive frames of facial expression images;
[0077] S1032, performing human body key point detection on the first image information to obtain a human body posture image, and determining human body posture time series data according to a plurality of consecutive frames of human body posture images;
[0078] S1033. Input the facial expression time series data and the human body posture time series data into a pre-trained speech interruption recognition model, and determine whether the target occupant has interrupted his speech based on the recognition result.
[0079] Specifically, face detection is performed on the first image information through a pre-trained face detection model, and a facial expression image is obtained based on the recognized face area, and multiple frames of facial expression images can form facial expression time series data; human key point detection is performed on the first image information through a pre-trained human key point detection model, and a human posture image can be obtained based on the positions of the recognized key points such as the head, upper torso, and hands, and multiple frames of human posture images can form human posture time series data; the facial expression time series data and the human posture time series data are input into a pre-trained speech interruption recognition model to determine whether the target occupant has interrupted his speech.
[0080] It should be noted that the face detection model and the human key point detection model of the embodiment of the present invention can adopt the existing neural network model in the prior art, and the embodiment of the present invention will not be described in detail here. The speech interruption recognition model of the embodiment of the present invention is obtained by collecting sample data formed by historical voice interaction data, and using a neural network to learn the association between the user's language expression intention and his facial expression and body posture. It can be recognized that when the user's thinking is interrupted, the user forgets the instructions that need to be issued for a while, or other things interrupt the speech, the user generally has expressions / actions such as keeping the mouth open, moving the eyeballs, looking up, holding the forehead with the hand, and grabbing the back of the head. The embodiment of the present invention learns the association between these expressions and actions and whether the user is interrupted in speaking through a neural network, thereby training a speech interruption recognition model, which is more advantageous for children with weaker language logic but richer expressions and behaviors.
[0081] As an optional implementation, the voice input waiting control method further includes the step of pre-training a speech interruption recognition model, which specifically includes:
[0082] S201, obtaining a plurality of preset speech interruption sample data, where the speech interruption sample data includes facial expression time series sample data and human body posture time series sample data;
[0083] S202, determining label information corresponding to each speech interruption sample data, and constructing a training data set according to the speech interruption sample data and the label information;
[0084] S203, inputting the training data set into a pre-built two-layer bidirectional recurrent neural network for training to obtain a trained speech interruption recognition model;
[0085] The two-layer bidirectional recurrent neural network includes an input layer, a first forward hidden layer, a second forward hidden layer, a first reverse hidden layer, a second reverse hidden layer and an output layer.
[0086] Specifically, when constructing the training data set, the facial expression time series sample data and human posture time series sample data of the historical occupants in the test vehicle when using the voice interaction function are obtained, and the acquisition method thereof refers to the aforementioned facial expression time series data and human posture time series data; at the same time, based on the actual situation (i.e., whether the historical occupants subsequently repeated the input) and manual annotation, the label information corresponding to each speech interruption sample data is determined, and the label information is used to indicate whether the historical occupant corresponding to each speech interruption sample data interrupted the speech or terminated the speech. The training data set can be generated according to the speech interruption sample data and the corresponding label information.
[0087] The speech interruption recognition model of the embodiment of the present invention is obtained by training a two-layer bidirectional recurrent neural network, which includes two hidden layers respectively used for feature extraction of facial expression time series data and human posture time series data, and the two hidden layers share an input layer and an output layer; each hidden layer is divided into a positive hidden layer and a negative hidden layer, which are respectively used for performing positive hidden state calculation and reverse hidden state calculation on the facial expression time series data / human posture time series data; after splicing the hidden state vectors obtained by the bidirectional hidden state calculation inside each hidden layer, the splicing vectors output by each hidden layer are weighted averaged based on a preset weighting parameter to obtain a final hidden state vector and obtain the corresponding speech interruption recognition result through the output layer output.
[0088] As an optional implementation, the training data set is input into a pre-built two-layer bidirectional recurrent neural network for training, which specifically includes:
[0089] S2031, inputting the speech interruption sample data into the input layer, calculating the facial expression time series sample data through the first forward hidden layer to obtain a first hidden state vector, and calculating the human body posture time series sample data through the second forward hidden layer to obtain a second hidden state vector;
[0090] S2032, inputting the speech interruption sample data into the input layer in reverse order, calculating the reversed facial expression time series sample data through the first reverse hidden layer to obtain a third hidden state vector, and calculating the reversed human body posture time series sample data through the second reverse hidden layer to obtain a fourth hidden state vector;
[0091] S2033, performing inverse processing on the third hidden state vector to obtain a fifth hidden state vector, and performing inverse processing on the fourth hidden state vector to obtain a sixth hidden state vector;
[0092] S2034, concatenating the first hidden state vector and the fifth hidden state vector to obtain a seventh hidden state vector, and concatenating the second hidden state vector and the sixth hidden state vector to obtain an eighth hidden state vector;
[0093] S2035, performing weighted averaging processing on the seventh hidden state vector and the eighth hidden state vector according to a preset weight parameter to obtain a ninth hidden state vector, and then inputting the ninth hidden state vector to an output layer to obtain a speech interruption prediction result;
[0094] S2036, determining a loss value of a two-layer bidirectional recurrent neural network according to the speech interruption prediction result and the label information;
[0095] S2037. Update the model parameters of the double-layer bidirectional recurrent neural network according to the loss value to obtain a trained speech interruption recognition model.
[0096] Specifically, the embodiment of the present invention can perform weight distribution on the facial expression time series data and the human posture time series data based on the attention mechanism. Through the weight distribution, the influence of different data on the speech interruption recognition result can be set, thereby improving the accuracy of speech interruption recognition. The specific weight distribution value can be preset. After the data in the training data set with weight parameters is input into the initialized two-layer bidirectional recurrent neural network, the recognition result output by the model, that is, the speech interruption prediction result, can be obtained. The accuracy of the model prediction can be evaluated according to the speech interruption prediction result and the aforementioned label information, thereby updating the parameters of the model. For the speech interruption recognition model, the accuracy of the model prediction result can be measured by the loss function. The loss function is defined on a single training data and is used to measure the prediction error of a training data. Specifically, the loss value of the training data is determined by the label of a single training data and the prediction result of the model for the training data. In actual training, a training data set has a lot of training data, so the cost function is generally used to measure the overall error of the training data set. The cost function is defined on the entire training data set and is used to calculate the average value of the prediction error of all training data, which can better measure the prediction effect of the model. For a general machine learning model, the aforementioned cost function, plus a regularization term that measures the complexity of the model, can be used as the objective function of the training, and the loss value of the entire training data set can be calculated based on the objective function. There are many types of commonly used loss functions, such as 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross entropy loss function, etc., which can all be used as loss functions of machine learning models, which will not be elaborated one by one here. In an embodiment of the present invention, any loss function can be selected to determine the loss value of the training. Based on the loss value of the training, the back propagation algorithm is used to update the parameters of the model, and a trained speech interruption recognition model can be obtained after several rounds of iteration. Specifically, the number of iterations can be set in advance, or the training is considered to be completed when the test set meets the accuracy requirements.
[0097] S104, when it is determined that the target occupant has interrupted his speech, obtaining the second voice information of the target occupant within a preset voice input waiting time, and then synthesizing the first voice information and the second voice information to obtain the target voice input information.
[0098] Specifically, when it is determined that the target occupant has interrupted speaking, the on-board voice recognition function is controlled to temporarily not make a judgment on invalid voice commands, and the on-board voice recognition function is kept in an activated state until a preset voice input waiting time (such as 5s, which can be set manually) is reached. During the period of time when the on-board voice recognition function is kept in an activated state, the second voice information of the occupant can continue to be obtained, and then the first voice information is synthesized with the second voice information to obtain the target voice input information.
[0099] As an optional implementation, when it is determined that the target occupant has interrupted his speech, the second voice information of the target occupant is obtained within a preset voice input waiting time, and then the target voice input information is synthesized according to the first voice information and the second voice information, which specifically includes:
[0100] S1041, when it is determined that the target occupant has interrupted speaking, keeping the vehicle-mounted voice recognition function in an activated state until the voice input waiting time is reached, and obtaining second voice information within the voice input waiting time;
[0101] S1042: Perform voice recognition and text concatenation on the first voice information and the second voice information to obtain target voice input information.
[0102] In some optional embodiments, when it is determined that the occupant has stopped speaking, or when the second voice information of the occupant has not been obtained when the preset voice input waiting time is reached, the vehicle-mounted voice recognition function is controlled to switch to a sleep state.
[0103] The above is a description of the method steps of the embodiment of the present invention. It can be recognized that the embodiment of the present invention obtains the image information of the passenger compartment while activating the voice recognition function to obtain the first voice information of the occupant, and determines whether the voice input is complete based on the first voice information. If the voice input is incomplete, the facial expression timing data and human posture timing data of the occupant are extracted based on the image information, and the facial expression timing data and human posture timing data are combined to determine whether the occupant has interrupted his speech. When it is determined that the occupant has interrupted his speech, the voice recognition function is kept in an activated state within the preset voice input waiting time, so that the occupant can continue to input the second voice information within the voice input waiting time. The complete voice input information can be synthesized based on the first voice information and the second voice information, and there is no need to re-enter the complete voice command after the voice input is interrupted, and there is no need to repeatedly wake up the voice recognition function, which improves the convenience of voice input and the user's driving experience.
[0104] Reference Figure 2 , an embodiment of the present invention provides a voice input waiting control system based on machine vision, comprising:
[0105] an information acquisition module, for acquiring first image information of a target passenger compartment and first voice information of a target passenger in response to a voice recognition activation instruction;
[0106] A first judgment module, used to judge whether the voice input is complete according to the first voice information;
[0107] A second judgment module is used to determine facial expression time series data and human posture time series data of the target occupant according to the first image information when determining that the voice input is incomplete, and then determine whether the target occupant is interrupted in speaking according to the facial expression time series data and human posture time series data;
[0108] The voice input waiting module is used to obtain the second voice information of the target occupant within a preset voice input waiting time when it is determined that the target occupant has interrupted speaking, and then synthesize the target voice input information according to the first voice information and the second voice information.
[0109] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0110] Reference Figure 3 , an embodiment of the present invention provides a voice input waiting control device based on machine vision, comprising:
[0111] at least one processor;
[0112] at least one memory for storing at least one program;
[0113] When the at least one program is executed by the at least one processor, the at least one processor implements the machine vision-based voice input waiting control method.
[0114] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0115] An embodiment of the present invention further provides a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the above-mentioned machine vision-based voice input waiting control method.
[0116] A computer-readable storage medium of an embodiment of the present invention can execute a machine vision-based voice input waiting control method provided by an embodiment of the method of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0117] The embodiment of the present invention also discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 The method shown.
[0118] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0119] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified to the contrary, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0120] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0121] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0122] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the above-mentioned program is printed, since the above-mentioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in other suitable ways as necessary, and then stored in a computer memory.
[0123] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0124] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0125] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0126] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A voice input waiting control method based on machine vision, characterized in that: The following steps are involved: In response to the voice recognition activation instruction, acquiring first image information of a target passenger compartment and acquiring first voice information of a target passenger; Determining whether the voice input is complete according to the first voice information; When it is determined that the voice input is incomplete, determining the facial expression time series data and the human posture time series data of the target occupant according to the first image information, and then judging whether the target occupant has interrupted speaking according to the facial expression time series data and the human posture time series data; When it is determined that the target occupant has interrupted speaking, second voice information of the target occupant is obtained within a preset voice input waiting time, and then the target voice input information is synthesized according to the first voice information and the second voice information.
2. The method for controlling voice input waiting based on machine vision according to claim 1, characterized in that: The step of acquiring first image information of a target passenger compartment and first voice information of a target passenger in response to a voice recognition activation instruction specifically includes: When the voice recognition activation instruction is detected, the vehicle-mounted voice recognition function is activated, and the image acquisition device and the voice acquisition device arranged in the target passenger compartment are activated; The first image information is acquired through the image acquisition device, and the first voice information is acquired through the voice acquisition device.
3. The method for controlling voice input waiting based on machine vision according to claim 1, characterized in that: The determining whether the voice input is complete according to the first voice information specifically includes: Performing voice recognition on the first voice information to obtain first text content; Performing semantic analysis on the first text content to obtain a plurality of candidate voice instructions, and matching the candidate voice instructions according to a preset voice instruction library; If the match is successful, it is determined that the voice input is complete; if the match is unsuccessful, it is determined that the voice input is incomplete.
4. The method for controlling voice input waiting based on machine vision according to claim 1, characterized in that: The step of determining facial expression time series data and human posture time series data of the target occupant according to the first image information, and then judging whether the target occupant has interrupted speaking according to the facial expression time series data and the human posture time series data, specifically includes: Performing face detection on the first image information to obtain a facial expression image, and determining the facial expression time series data according to a plurality of consecutive frames of the facial expression images; Performing human body key point detection on the first image information to obtain a human body posture image, and determining the human body posture time series data according to a plurality of consecutive frames of the human body posture images; The facial expression time series data and the human body posture time series data are input into a pre-trained speech interruption recognition model, and whether the target occupant interrupts speech is determined according to the recognition result.
5. The method for controlling voice input waiting based on machine vision according to claim 4, characterized in that: The voice input waiting control method further includes the step of pre-training the speech interruption recognition model, which specifically includes: Acquire a plurality of preset speech interruption sample data, wherein the speech interruption sample data includes facial expression time series sample data and human body posture time series sample data; Determining label information corresponding to each of the speech interruption sample data, and constructing a training data set according to the speech interruption sample data and the label information; Inputting the training data set into a pre-built two-layer bidirectional recurrent neural network for training to obtain the trained speech interruption recognition model; The two-layer bidirectional recurrent neural network includes an input layer, a first forward hidden layer, a second forward hidden layer, a first reverse hidden layer, a second reverse hidden layer and an output layer.
6. The method for controlling voice input waiting based on machine vision according to claim 5, characterized in that: The step of inputting the training data set into a pre-built two-layer bidirectional recurrent neural network for training specifically includes: Inputting the speech interruption sample data into the input layer, calculating the facial expression time series sample data through the first forward hidden layer to obtain a first hidden state vector, and calculating the human body posture time series sample data through the second forward hidden layer to obtain a second hidden state vector; Inputting the speech interruption sample data into the input layer in reverse order, calculating the third hidden state vector on the reversed facial expression time series sample data through the first reverse hidden layer, and calculating the fourth hidden state vector on the reversed human body posture time series sample data through the second reverse hidden layer; Reverse processing is performed on the third hidden state vector to obtain a fifth hidden state vector, and reverse processing is performed on the fourth hidden state vector to obtain a sixth hidden state vector; Concatenate the first hidden state vector and the fifth hidden state vector to obtain a seventh hidden state vector, and concatenate the second hidden state vector and the sixth hidden state vector to obtain an eighth hidden state vector; Performing weighted averaging processing on the seventh hidden state vector and the eighth hidden state vector according to a preset weight parameter to obtain a ninth hidden state vector, and then inputting the ninth hidden state vector into the output layer to obtain a speech interruption prediction result; Determining a loss value of the two-layer bidirectional recurrent neural network according to the speech interruption prediction result and the label information; The model parameters of the two-layer bidirectional recurrent neural network are updated according to the loss value to obtain the trained speech interruption recognition model.
7. A method for controlling voice input waiting based on machine vision according to any one of claims 1 to 6, characterized in that: When it is determined that the target occupant has interrupted speaking, obtaining second voice information of the target occupant within a preset voice input waiting time, and then synthesizing the first voice information and the second voice information to obtain target voice input information, which specifically includes: When it is determined that the target occupant has interrupted speaking, keeping the vehicle-mounted voice recognition function in an activated state until the voice input waiting time is reached, and acquiring the second voice information within the voice input waiting time; Perform voice recognition and text concatenation on the first voice information and the second voice information to obtain the target voice input information.
8. A voice input waiting control system based on machine vision, characterized in that: include: an information acquisition module, for acquiring first image information of a target passenger compartment and first voice information of a target passenger in response to a voice recognition activation instruction; A first judgment module, used for judging whether the voice input is complete according to the first voice information; A second judgment module is used to determine the facial expression time series data and the human posture time series data of the target occupant according to the first image information when it is determined that the voice input is incomplete, and then determine whether the target occupant is interrupted in speaking according to the facial expression time series data and the human posture time series data; The voice input waiting module is used to obtain the second voice information of the target occupant within a preset voice input waiting time when it is determined that the target occupant has interrupted speaking, and then synthesize the first voice information and the second voice information to obtain the target voice input information.
9. A voice input waiting control device based on machine vision, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a machine vision-based voice input waiting control method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to execute a machine vision-based voice input waiting control method as claimed in any one of claims 1 to 7 when executed by the processor.
Citation Information
Patent Citations
Voice detection method and device, prediction model training method and device, equipment and medium
CN112567457A
Emotion recognition method, system and device based on multi-modal feature fusion and medium
CN113807249A
Vehicle control method, training method and device, vehicle, server and medium
CN116645959A
Speech recognition method and device, equipment and storage medium
CN117219074A
KR20210040913A
