In-vehicle voice recognition processing method and device, electronic equipment and medium

Abstract

The invention relates to an in-vehicle voice recognition processing method and device, electronic equipment and a medium, and the method comprises the steps: obtaining mixed audio information and video information collected by a vehicle-mounted system, carrying out the voice separation of the mixed audio information, obtaining at least one piece of sub-audio information, determining a target user corresponding to each piece of sub-audio information, and carrying out the recognition processing of the in-vehicle voice based on the target user. Performing user matching in combination with the video information to obtain image state information corresponding to the target user, then generating a user communication picture corresponding to the vehicle-mounted system according to the image state information and in combination with the sub-audio information corresponding to the target user, and performing output according to the user communication picture; therefore, the in-vehicle communication condition can be accurately displayed for the user in real time through the user communication picture, the communication experience of the user in the vehicle is effectively improved, and the problem that the communication experience of the user in the vehicle is poor due to the fact that the user cannot accurately obtain the communication content in the vehicle in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of automobile technology, and in particular to a method, device, electronic equipment and medium for recognizing and processing in-vehicle speech. Background Art

[0002] With the development of vehicles, vehicle travel has gradually become the main means of people's daily life. Since vehicle travel is often accompanied by communication between users in the car, if there are multiple users communicating in the car, there will be not only user voice communication in the car, but also various noises such as media music, navigation broadcast, voice assistant interaction sound, vehicle driving noise, etc., which makes it impossible for users to accurately obtain the communication content and it is very easy to miss the communication content, resulting in a poor communication experience for users in the car. Summary of the invention

[0003] The present application provides a method, device, electronic device and medium for recognizing and processing in-vehicle speech to solve the problem of poor in-vehicle user communication experience caused by the inability of users to accurately obtain the in-vehicle communication content in the existing related technologies.

[0004] In a first aspect, the present application provides a method for recognizing and processing in-vehicle speech, comprising:

[0005] Obtain mixed audio and video information in the car;

[0006] Performing voice separation on the mixed audio information to obtain sub-audio information;

[0007] Determine a target user corresponding to each piece of the sub-audio information according to the sub-audio information and / or the video information;

[0008] Based on the target user, user matching is performed in combination with the video information to obtain image state information corresponding to the target user, where the image state information is the riding state of the target user in the car;

[0009] A user communication screen is generated according to the image state information and the sub-audio information corresponding to the target user, and the user communication screen is output and displayed.

[0010] Optionally, determining a target user corresponding to each piece of the sub-audio information according to the sub-audio information and / or the video information includes:

[0011] Obtaining a preset voice feature corresponding to at least one preset user;

[0012] Extracting user voice features from each piece of sub-audio information;

[0013] Comparing the user's voice features with various preset voice features to obtain a voice feature comparison result;

[0014] In the case where the voice feature comparison result is a feature matching result, a preset user corresponding to the voice feature comparison result is determined as a target user.

[0015] Optionally, the in-vehicle voice recognition and processing method further includes:

[0016] When the speech feature comparison result is a feature mismatch result, extracting audio time information from the sub-audio information;

[0017] Extracting, from the video information, a target video frame that matches the audio time information in time;

[0018] The target user corresponding to the sub-audio information is determined by combining the user voice feature and the target video frame.

[0019] Optionally, performing user matching based on the target user and in combination with the video information to obtain image state information corresponding to the target user includes:

[0020] Performing user identification on the video information to obtain an image user;

[0021] Matching the target user with the image user to obtain a matching result;

[0022] When the matching result is a user matching result, extracting user status information corresponding to the image user from the video information;

[0023] The user status information is determined as the image status information corresponding to the target user.

[0024] Optionally, matching the target user with the image user to obtain a matching result includes:

[0025] In the case where the target user is a preset user, obtaining first physiological characteristic information corresponding to the target user, and obtaining second physiological characteristic information corresponding to the image user;

[0026] Matching the first physiological characteristic information with the second physiological characteristic information to obtain a physiological characteristic result;

[0027] The matching result is generated according to the physiological characteristic result.

[0028] Optionally, matching the target user with the image user to obtain a matching result includes:

[0029] When the target user does not belong to the preset user, obtaining the first physiological characteristic information and the sub-audio information corresponding to the target user;

[0030] Acquire second physiological characteristic information and lip movement information corresponding to the image user according to the video information;

[0031] Matching the first physiological characteristic information with the second physiological characteristic information to obtain a physiological characteristic result;

[0032] Matching the speech text corresponding to the sub-audio information with the lip movement information to obtain a lip shape matching result;

[0033] The matching result is obtained according to the physiological feature result and the lip shape matching result.

[0034] Optionally, outputting and displaying the user communication screen includes:

[0035] Determine a user location and a target display screen identifier corresponding to each of the target users;

[0036] Adjusting the user communication screen according to the user position to obtain a display output screen;

[0037] The target display screen corresponding to the target display screen identifier is used to output and display the display output screen.

[0038] Optionally, after performing voice separation on the mixed audio information to obtain a plurality of sub-audio information, the method further includes:

[0039] Extracting keywords from each of the sub-audio information to obtain user keywords;

[0040] In the case where the user keyword belongs to a preset control keyword, control execution prompt information is output.

[0041] In a second aspect, a device for recognizing and processing in-vehicle speech is provided, comprising:

[0042] An acquisition module, used for acquiring mixed audio information and video information in the vehicle;

[0043] A speech separation module, used for performing speech separation on the mixed audio information to obtain sub-audio information;

[0044] a determination module, configured to determine a target user corresponding to each piece of the sub-audio information according to the sub-audio information and / or the video information;

[0045] A user matching module, configured to perform user matching based on the target user and the video information, and obtain image status information corresponding to the target user, wherein the image status information is the riding status of the target user in the vehicle;

[0046] A generating module is used to generate a user communication screen according to the image state information and the sub-audio information corresponding to the target user, and output and display the user communication screen.

[0047] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0048] Memory, used to store computer programs;

[0049] The processor is used to implement the in-vehicle speech recognition and processing method described in any one of the first aspects when executing the program stored in the memory.

[0050] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for recognizing and processing in-vehicle speech as described in any one of the first aspects is implemented.

[0051] In a fourth aspect, a vehicle is provided, wherein the vehicle comprises the in-vehicle speech recognition and processing device as described in the second aspect.

[0052] The embodiment of the present application obtains mixed audio information and video information in the car, separates the speech from the mixed audio information, and obtains sub-audio information; then determines the target user corresponding to each sub-audio information according to the sub-audio information and / or video information; and performs user matching based on the target user in combination with the video information to obtain image status information corresponding to the target user, where the image status information is the riding status of the target user in the car; generates a user communication screen according to the image status information and the sub-audio information corresponding to the target user, and then outputs and displays the user communication screen; enables the users in the car to obtain the communication content in real time and accurately through the user communication screen, so that the users in the car can understand the communication situation in the car in real time and accurately, thereby solving the problem of poor communication experience of users in the car caused by the inability of users in the car to accurately obtain the communication content in the existing related technology, and can effectively improve the communication experience of users in the car. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A flowchart of a method for recognizing and processing in-vehicle speech provided in an embodiment of the present application;

[0054] Figure 2 Another flowchart of a method for recognizing and processing in-vehicle speech provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of the structure of a vehicle voice recognition and processing device provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0058] When multiple users are traveling by car, in-car communication often occurs. However, the in-car environment is relatively complex, with multiple audios mixed in, such as user communication voices, media music, navigation broadcasts, voice assistant interaction sounds, and vehicle driving noise. As a result, it is inevitable that users cannot hear clearly what other users are saying during communication. At this time, if the communication is frequently interrupted to inquire about missed content, it will lead to a poor in-car user communication experience.

[0059] In order to solve the problem of poor in-car user communication experience in the existing related technologies, the present application provides a method, device, electronic device and medium for recognizing and processing in-car voice, by obtaining mixed audio information and video information in the car, performing voice separation on the mixed audio information to obtain sub-audio information; then determining the target user corresponding to each sub-audio information according to the sub-audio information and / or video information; and based on the target user, performing user matching in combination with the video information to obtain image status information corresponding to the target user, the image status information being the riding status of the target user in the car; generating a user communication screen according to the image status information and the sub-audio information corresponding to the target user, and then outputting and displaying the user communication screen; allowing in-car users to obtain the communication content in real time and accurately through the user communication screen, making it convenient for in-car users to understand the in-car communication situation in real time and accurately, thereby solving the problem of poor in-car user communication experience caused by the inability of in-car users to accurately obtain the communication content in the existing related technologies, and can effectively improve the in-car user communication experience.

[0060] Figure 1A flow chart of a method for recognizing and processing in-vehicle speech provided in an embodiment of the present application. The method can be applied to an execution subject such as a vehicle, an on-board system, etc. that has vehicle information collection. In addition, the execution subject of the method can be hardware or software. When the above-mentioned execution subject is hardware, the execution subject can be one or more of the above-mentioned electronic devices. For example, a single electronic device can execute the method, or multiple electronic devices can cooperate with each other to execute the method. When the above-mentioned execution subject is software, the method can be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0061] like Figure 1 As shown, a method for recognizing and processing in-vehicle speech provided in an embodiment of the present application may specifically include the following steps:

[0062] Step S110: Acquire mixed audio information and video information in the car;

[0063] Among them, the mixed audio information can represent the in-car audio information collected by an audio collection device pre-configured in the car, and the preset audio collection device can be a microphone and other devices; and the video information can represent the in-car video collected by the vehicle-mounted system through a video collection device pre-configured in the car, and the preset image collection device can be a camera or scanning device and other devices.

[0064] In addition, the specific acquisition of mixed audio information and video information data in the car can be performed by the vehicle-mounted system as the execution entity. The vehicle-mounted system can represent a system configured in the vehicle. The vehicle-mounted system can include but is not limited to systems with functions such as in-vehicle information collection, information provision, entertainment, navigation, communication and vehicle control. This embodiment does not make specific limitations on this.

[0065] Step S120: performing speech separation on the mixed audio information to obtain sub-audio information.

[0066] Specifically, after obtaining the mixed audio information, the mixed audio information can be subjected to speech separation to obtain at least one sub-audio information, wherein speech separation may include but is not limited to feature extraction, audio separation and other processing, so that the sub-audio information can represent the speech emitted by a specific user, that is, for each speech emitted by a user contained in the mixed audio, a sub-audio information can be obtained through speech separation.

[0067] It should be noted that, after performing voice separation on the mixed audio information, this embodiment can determine whether at least one sub-audio information is obtained. If at least one sub-audio information is not obtained, it means that the current mixed audio information does not contain the voice emitted by the user, and there is no need to execute subsequent steps. If at least one sub-audio information is obtained, it means that the current mixed audio information contains the voice emitted by the user, and subsequent steps can be executed to generate a user communication screen corresponding to the vehicle-mounted system and output it according to the user communication screen.

[0068] Specifically, the specific method of performing speech separation on the mixed audio information to obtain at least one sub-audio information can use a pre-configured audio separation algorithm for speech separation and separation, such as blind source separation (Blind Source Separation) or a deep learning model to separate the mixed audio into sub-audio information of each object. The model used can be a sound source separation model based on a deep neural network or an attention mechanism, which can distinguish the voices of different people in the mixed audio. Among them, the blind source separation algorithm can use independent component analysis (ICA) or non-negative matrix factorization (NMF).

[0069] Independent Component Analysis (ICA) is used to extract the individual audio of each object from the mixed audio of multiple objects. The specific steps can be referred to as follows:

[0070] Preprocessing: First, preprocess the mixed audio, such as removing noise, performing speech enhancement, etc.

[0071] Feature extraction: The preprocessed mixed audio is converted into a time-frequency domain feature representation, and the audio signal is converted into a spectrogram using Short-Time Fourier Transform (STFT).

[0072] ICA model fitting: Apply the independent component analysis (ICA) algorithm to fit the ICA model on the feature representation. ICA assumes that the mixed audio is a linear combination of multiple independent audio signals. By estimating the inverse mixing matrix, the mixed audio can be separated into independent audio signals.

[0073] Audio separation: Using the estimated inverse mixing matrix, the ICA model is applied to the feature representation of the mixed audio, and an inverse transform is performed to restore the audio signal in the time domain to obtain the sub-audio information.

[0074] Of course, the above is only an example to illustrate the effect, and this embodiment does not make any specific limitation to this.

[0075] Step S130: determining a target user corresponding to each piece of sub-audio information according to the sub-audio information and / or the video information.

[0076] Specifically, after obtaining the sub-audio information, the target user corresponding to each sub-audio information can be determined based on the sub-audio information and / or video information, where the target user can be used to represent the user who sends the sub-audio information; it should be noted that the target user in this embodiment can be a pre-configured preset user, or it can be an identified user determined in real time based on the sub-audio information, and this embodiment does not make any specific limitations on this.

[0077] In addition, in the present embodiment, in the process of determining the target user corresponding to each sub-audio information, the determination can be made in combination with the sub-audio information and / or the video information. For example, voice features can be extracted from the sub-audio information and the target user can be identified using the voice features. User analysis can also be performed through image users in the time-frequency information to determine the target user corresponding to the sub-audio information. Of course, the target user corresponding to each sub-audio information can also be determined by combining the sub-audio information and the video information for feature matching, etc. This embodiment does not specifically limit this.

[0078] Step S140: Based on the target user, user matching is performed in combination with the video information to obtain image status information corresponding to the target user, where the image status information is the riding status of the target user in the car.

[0079] Specifically, after determining the target user, user matching can be performed based on the target user and combined with the video information to obtain image status information corresponding to the target user, where user matching can represent searching for a user matching the target user in the video information, and the image status information can represent the riding status of the target user in the car. The riding status can include but is not limited to the riding position, riding posture, etc., and this embodiment does not make specific limitations on this.

[0080] Specifically, in the process of user matching, the features corresponding to each image user in the video information can be identified, the features corresponding to each image user in the video information can be compared with the features corresponding to the target user to determine whether they match, the image user that matches the features corresponding to the target user can be determined as a matching user, and the image status information corresponding to the matching user can be extracted from the video information.

[0081] Step S150: generating a user communication screen according to the image state information and the sub-audio information corresponding to the target user, and outputting and displaying the user communication screen.

[0082] Specifically, after obtaining the image status information corresponding to the target user, a user communication screen can be generated according to the image status information and the sub-audio information corresponding to the target user, and the user communication screen can be output and displayed. The user communication screen can represent a specific screen containing the communication between each target user in the current vehicle, that is, the sub-audio information and image status information corresponding to each target user can be displayed through the user communication screen; wherein, the method of displaying the sub-audio information can be displayed through the text of a specific dialog box, so as to achieve the purpose of displaying that each target user communicates by speaking the corresponding text of the voice; the method of displaying the image status information can be displayed through a specific shape pattern, so as to achieve the purpose of displaying the riding status of each target user; of course, other methods can also be used to display it in the user communication screen, and this embodiment does not make specific restrictions on this. After obtaining the user communication screen, the user communication screen can be output, and the specific output method can be output through the vehicle main control display screen, or it can be split-screen displayed according to the location of each target user. Of course, other output methods can also be used, and this embodiment does not make specific restrictions on this.

[0083] It can be seen that this embodiment obtains the mixed audio information and video information collected by the vehicle-mounted system, performs voice separation on the mixed audio information, obtains multiple sub-audio information, determines the target user corresponding to each sub-audio information, and performs user matching based on the target user in combination with the video information to obtain the image status information corresponding to the target user, the image status information is the riding status of the target user in the car, and then generates a user communication screen according to the image status information and the sub-audio information corresponding to the target user, and outputs and displays the user communication screen. That is, this embodiment can not only display the sub-audio information output during the communication process of each target user through the user communication screen, but also enable the in-car users to obtain the communication content in real time and accurately through the user communication screen, which is convenient for the in-car users to understand the in-car communication situation in real time and accurately, thereby solving the problem of poor in-car user communication experience caused by the inability of in-car users to accurately obtain the communication content in the existing related technology, and can effectively improve the in-car user communication experience.

[0084] like Figure 2 As shown, in an optional embodiment of the present application, step S130 determines the target user corresponding to each piece of sub-audio information according to the sub-audio information and / or the video information, which may specifically include the following steps:

[0085] Step S131: Acquire a preset voice feature corresponding to at least one preset user;

[0086] Step S132: extracting user voice features from each of the sub-audio information;

[0087] Step S133: when the voice feature comparison result is a feature matching result, determining the preset user corresponding to the voice feature comparison result as the target user;

[0088] After obtaining the sub-audio information, this embodiment can obtain the preset voice feature corresponding to at least one preset user, wherein the preset user can represent the user who is pre-recorded and configured, and the preset voice feature can represent the voice feature corresponding to the preset user; the user voice feature can be extracted from each sub-audio information, and the user voice feature can represent the voice feature contained in the sub-audio information. The voice feature can include but is not limited to voiceprint, fundamental frequency, formant, spectrum, etc., which can be used to describe and analyze the attribute characteristics of the voice signal, and this embodiment does not make specific restrictions on this. After obtaining the preset voice feature and the user voice feature, the user voice feature can be compared with each preset voice feature to obtain the voice feature comparison result, and the voice feature comparison result can be used to indicate whether the user voice feature matches each preset voice feature, that is, the voice feature comparison result can be a feature mismatch result or a feature match result, and the feature mismatch result is used to indicate the result that the user voice feature does not match each preset voice feature, and the feature match result is used to indicate the result that the user voice feature matches the preset voice feature. Of course, it can also be other representation methods, and this embodiment does not make specific restrictions on this. When the voice feature comparison result is a feature matching result, it means that there is a preset voice feature that matches the user voice feature. At this time, the preset user corresponding to the preset voice feature that matches the user voice feature can be determined as the target user.

[0089] like Figure 2 As shown, in an optional embodiment of the present application, the method for recognizing and processing in-vehicle speech may further include the following sub-steps:

[0090] Step S134: when the voice feature comparison result is a feature mismatch result, extracting audio time information from the sub-audio information;

[0091] Step S135: extracting a target video frame that matches the audio time information from the video information;

[0092] Step S136: Determine the target user corresponding to the sub-audio information by combining the user voice features and the target video frame.

[0093] After obtaining the voice feature comparison result, this embodiment can determine whether the voice feature comparison result is a feature mismatch result. When the voice feature comparison result is a feature mismatch result, it indicates that the audio time information is extracted from the sub-audio information at this time, and the audio time information represents the time interval spanned by the sub-audio information, such as 18:01:05-18:01:10; then, the target video frame that matches the audio time information can be extracted from the video information, and the target video frame can represent the video frame corresponding to the time interval represented by the sub-audio information, that is, the target video frame can contain multiple video frames; then, the user voice feature and the target video frame can be combined to determine the target user corresponding to the sub-audio information, wherein the user voice feature information can be matched with the image user in the target video frame, and the specific matching method can be lip shape matching, physiological feature matching, etc., which is not specifically limited in this embodiment; thereby, the target user that meets the user voice feature can be determined from the target video frame.

[0094] It should be noted that in the process of combining the user voice features and the target video frame to determine the target user corresponding to the sub-audio information, feature recognition can be performed based on the user voice features to obtain the first physiological feature information. The first physiological feature information can be used to represent the physiological features corresponding to the user who sends the sub-audio information, such as gender features, age features, etc. Then, user analysis is performed on the target video frame to obtain the second physiological feature information. The second physiological feature information can represent the physiological features corresponding to each user contained in the image information, so that the first physiological feature information can be matched with the second physiological feature information, and the user corresponding to the second physiological feature information that matches the first physiological feature information can be determined as the target user. Subsequently, the target user can be stored and recorded as a preset user, so as to achieve the purpose of improving the recognition efficiency and recognition accuracy of subsequent re-identification.

[0095] As an example of the present application, there are three target users communicating in a car, where the image status information corresponding to target user A is male and is driving the vehicle, and the corresponding sub-audio information is "Son, close the window, haha"; the image status information corresponding to target user B is female and is in the co-pilot seat with the body facing left, and the corresponding sub-audio information is "Where did I put that notebook?"; the image status information corresponding to target user C is male and is in the right side of the back row with the body facing right, and the corresponding sub-audio information is "Let me take another look outside"; at this time, the physiological characteristic information corresponding to target user A is a male adult, the image status information is driving the vehicle, and the sub-audio information is "Son, close the window, haha"; the physiological characteristic information corresponding to target user A is a female adult, the image status information is facing the left side, and the sub-audio information is "Where did I put that notebook?"; the physiological characteristic information corresponding to target user C is a male child, the image status information is facing the right side, and the sub-audio information is "Let me take another look outside"; at this time, based on The image status information is combined with the sub-audio information corresponding to the target user to generate the user communication screen corresponding to the vehicle system. The various seat areas can be divided according to the in-vehicle space layout. For example, a large blue human-shaped image is displayed in the driver's seat area and faces forward holding the steering wheel. The large blue human-shaped image displays the text corresponding to "Son, close the window" on the side; a large pink human-shaped image is displayed in the co-pilot area and faces the left. The large pink human-shaped image displays the text corresponding to "Where did I put my notebook?" on the side; a small blue human-shaped image is displayed in the right rear area and faces the left. The small blue human-shaped image displays the text corresponding to "Let me take another look outside" on the side; thus, the large blue human-shaped image can be used to indicate that the physiological characteristic information is a male adult, the large pink human-shaped image can be used to indicate that the physiological characteristic information is a female adult, and the small blue human-shaped image can be used to indicate that the physiological characteristic information is a male child; and in the above examples, text is used to display the sub-audio information, and the specific shape of the human-shaped image is used to indicate different image status information, such as the orientation of the human-shaped image. Therefore, the in-car communication situation can be displayed to the user in real time and accurately through the user communication screen, thereby effectively improving the in-car user communication experience, and solving the problem of poor in-car user communication experience caused by the inability of users to accurately obtain the in-car communication content in the existing related technologies; in addition, the riding status of the rear-seat users can also be checked through the user communication screen, avoiding the need for users in the driving seat or the co-pilot seat to frequently look back or turn sideways to check the riding status of the rear-seat users, which affects the driver's concentration on driving the vehicle and affects driving safety, and can achieve the purpose of effectively improving driving safety.

[0096] In an optional embodiment of the present application, step S140 performs user matching based on the target user in combination with the video information to obtain image status information corresponding to the target user, which may specifically include the following steps: performing user identification on the video information to obtain the image user; matching the target user with the image user to obtain a matching result; when the matching result is a user matching result, extracting user status information corresponding to the image user from the video information; and determining the user status information as the image status information corresponding to the target user.

[0097] After determining the target user, this embodiment can perform user identification on the video information to obtain the image user. User identification can refer to identifying the user in the image represented by the video information, which can specifically include but not limited to human body identification, physiological feature identification, etc., and the user identified from the video information is determined as the image user. Of course, one or more image users can be identified; thereby, the target user can be matched with each image user to obtain a matching result, wherein the matching process can include but not limited to physiological feature matching, facial feature matching, etc., and the matching result can be used to indicate whether the target user matches the image user, that is, the matching result can be used to indicate whether the target user matches the image user. The user matching result may be a user mismatching result indicating that the target user does not match the image user. When the matching result is a user matching result, it indicates that the target user and the image user are the same user. At this time, the user status information corresponding to the image user may be extracted from the video information. The user status information may indicate the current status of the image user, and may specifically include the riding position, riding status, etc. The user status information may then be determined as the image status information corresponding to the target user. Specifically, the user status information may be directly determined as the image status information, or the user status information may be converted into image status information. This embodiment does not specifically limit this.

[0098] In one example, user identification is performed on video information, and the image users obtained include image user A, image user B, and image user C. The target user is matched with image user A to obtain matching result A, the target user is matched with image user B to obtain matching result B, and the target user is matched with image user C to obtain matching result C. When matching result A indicates that the user does not match, matching result B indicates that the user matches, and matching result C indicates that the user does not match, user status information corresponding to image user B can be extracted from the video information. The user status information can indicate that image user B is in the left position of the back row, facing the left window sideways, and the image status information is generated based on the user status information. Of course, the above is only an example for illustration, and this embodiment does not make specific limitations on this.

[0099] In an optional embodiment of the present application, the target user is matched with the image user to obtain a matching result, which may specifically include the following steps: when the target user is a preset user, obtaining first physiological characteristic information corresponding to the target user, and obtaining second physiological characteristic information corresponding to the image user; matching the first physiological characteristic information with the second physiological characteristic information to obtain a physiological characteristic result; and generating a matching result based on the physiological characteristic result.

[0100] In the process of matching the target user with the image user and obtaining the matching result, this embodiment can determine whether the target user belongs to the preset user; if the target user belongs to the preset user, it means that the physiological characteristics corresponding to the target user have been pre-recorded, and the first physiological characteristic information corresponding to the target user can be obtained at this time. Since the first physiological characteristic information obtained at this time may include but not limited to age characteristics, gender characteristics, and facial characteristics, etc., the second physiological characteristic information corresponding to the image user can be obtained. The second physiological characteristic information can represent the physiological characteristics corresponding to the image user, such as age characteristics, gender characteristics, and facial characteristics, etc., so that the first physiological characteristic information can be matched with the second physiological characteristic information to obtain a physiological characteristic result. The result can be used to indicate whether the first physiological characteristic information matches the second physiological characteristic information. For example, the physiological characteristic result can be divided into a physiological characteristic mismatch result and a physiological characteristic match result, wherein the physiological characteristic mismatch result is a result indicating that the first physiological characteristic information does not match the second physiological characteristic information, and the physiological characteristic match result is a result indicating that the first physiological characteristic information matches the second physiological characteristic information. Furthermore, a matching result can be generated based on the physiological characteristic result. For example, when the physiological characteristic result is a physiological characteristic mismatch result, a user mismatch matching result can be generated. When the physiological characteristic result is a physiological characteristic match result, a user match matching result can be generated. This embodiment does not specifically limit this.

[0101] It can be seen that in this embodiment, when the target user belongs to the preset user, the first physiological characteristic information corresponding to the target user can be directly matched with the second physiological characteristic information corresponding to the image user, so as to achieve the purpose of finding the user matched by the target user among the image users contained in the video information. In addition, in the specific process of matching the first physiological characteristic information with the second physiological characteristic information to obtain the physiological characteristic result, due to whether the target user belongs to the preset user, the first physiological characteristic information and the second physiological characteristic information can include but are not limited to age characteristics, gender characteristics and face characteristics, etc., and the matching method used at this time can include but are not limited to one or more matching methods such as age characteristic matching, gender characteristic matching and face characteristic matching, and generate the physiological characteristic result in combination with the feature matching results corresponding to each matching method, for example, only when all feature matching results are feature matching success results, the physiological characteristic matching result is generated, or when each feature matching result meets the preset conditions, the physiological characteristic matching result is generated, and this embodiment does not make specific restrictions on this. Since the matching composition of this embodiment includes face feature matching, it can effectively improve the accuracy and efficiency of matching the target user with the image user.

[0102] In one example, when the target user is a preset user, based on the image state information and the sub-audio information corresponding to the target user, in the generated user communication screen, the user tag corresponding to the preset user can be added to the corresponding position area. For example, when there are three users communicating in the car, the generated user communication screen can include the driver's seat area, the co-pilot area, and the right side area of ​​the rear row. The driver's seat area is used to display the corresponding image of the target user as "father" and the text of the sub-audio information corresponding to "father". The co-pilot area is used to display the corresponding image of the target user as "mother" and the text of the sub-audio information corresponding to "mother". The right side area of ​​the rear row is used to display the corresponding image of the target user as "son", such as displaying an image of the "son" leaning against the right side window, and the text of the sub-audio information corresponding to "son". Of course, the above is only an example for illustration, and this embodiment does not make specific limitations on this.

[0103] In an optional embodiment of the present application, the target user is matched with the image user to obtain a matching result, which may specifically include the following steps: when the target user does not belong to the preset user, obtaining the first physiological characteristic information and sub-audio information corresponding to the target user; based on the video information, obtaining the second physiological characteristic information and lip movement information corresponding to the image user; matching the first physiological characteristic information with the second physiological characteristic information to obtain a physiological characteristic result; matching the speech text and lip movement information corresponding to the sub-audio information to obtain a lip shape matching result; obtaining a matching result based on the physiological characteristic result and the lip shape matching result.

[0104] In the process of matching the target user with the image user and obtaining the matching result, this embodiment can determine whether the target user belongs to the preset user. If the target user does not belong to the preset user, it means that the physiological characteristics corresponding to the target user have not been pre-recorded. At this time, the first physiological characteristic information and sub-audio information corresponding to the target user can be obtained; since the video information can include one or more captured images, for example, all video frames of the in-vehicle video captured by the vehicle-mounted system, the second physiological characteristic information and lip movement information corresponding to the image user can be obtained based on the video information, wherein the lip movement information can represent the lip movement corresponding to the target user between multiple captured images; the first physiological characteristic information and the second physiological characteristic information are matched to obtain the physiological characteristic result. The specific matching method is the same as the type of the aforementioned embodiment and will not be repeated here; and text recognition can also be performed on the sub-audio information to obtain the voice text corresponding to the sub-audio information. The voice text can represent the text content corresponding to the sub-audio information. Then, the speech text and lip movement information corresponding to the sub-audio information can be matched to obtain a lip shape matching result, wherein a specific matching method can be to use a neural network or a preset model to compare and match the text and the lip shape, and the lip shape matching result can indicate whether the speech text and the lip shape represented by the lip movement information match, that is, the lip shape matching result can be a lip shape mismatch result indicating that the lip shapes represented by the speech text and the lip movement information do not match, or a lip shape matching result indicating that the lip shapes represented by the speech text and the lip movement information match; then, a matching result can be obtained based on the physiological feature result and the lip shape matching result, and a specific method for determining the matching result can be to determine the user matching result as the matching result when the physiological feature result is a physiological feature matching result and the lip shape matching result is a lip shape matching result, and to determine the user mismatch result as the matching result when the physiological feature result is a physiological feature mismatch result and / or the lip shape matching result is a lip shape mismatch result.

[0105] It can be seen that in this embodiment, when the target user is a pre-user, the first physiological characteristic information obtained at this time can only include physiological characteristics extracted by feature extraction of sub-audio information, such as age characteristics, gender characteristics, etc., and cannot include facial features, that is, facial feature matching cannot be performed subsequently; this embodiment, by adding the matching process of voice text and lip movement information, can achieve the purpose of improving the accuracy and efficiency of matching the target user with the image user by matching the voice text and lip movement information when facial feature matching cannot be performed.

[0106] In an optional embodiment of the present application, step S150 is output based on the user communication screen, which may specifically include the following steps: determining the user position and target display screen identifier corresponding to each target user; adjusting the user communication screen according to the user position to obtain a display output screen; and outputting the display output screen through the target display screen corresponding to the target display screen identifier.

[0107] Since the present embodiment can determine one or more target users, the user position and target display screen identifier corresponding to each target user can be determined in the process of outputting according to the user communication screen. The user position can indicate the specific position of the target user in the vehicle, and the target display screen identifier can indicate the pre-configured display screen identifier corresponding to the user position; thus, the user communication screen can be adjusted according to the user position to obtain a display output screen, and the display output screen can represent the screen for output to the target user for viewing, and the specific adjustment method can be to adjust the local screen at the user position in the user communication screen, so that the user can intuitively and quickly determine the content displayed on the current display output screen; and then the display output screen can be output through the target display screen corresponding to the target display screen identifier.

[0108] In one example, when the target users include target user A, target user B, and target user C, it is determined that the user position corresponding to target user A is the driving seat and the target display screen identifier is the main driving display screen identifier, the user position corresponding to target user B is determined to be the co-pilot seat and the target display screen identifier is the co-pilot display screen identifier, and the user position corresponding to target user C is determined to be the rear left seat and the target display screen identifier is the rear left display screen identifier; when the user communication screen is adjusted according to the user position to obtain the display output screen, the target user C is taken as an example for explanation. At this time, according to the user position, that is, the rear left seat, the user communication screen is adjusted to obtain the display output screen. The specific adjustment method can be to darken the partial screen of the rear left seat in the user communication screen, add an identification wireframe or remove the partial screen, add the text "your location" and the like to highlight the area of ​​the rear left seat, so that the rear left display screen corresponding to the rear left display screen identifier is output, and the target user C can quickly and accurately determine his current position from the display output screen, so that the target user C can quickly view the text corresponding to the voice spoken by other target users during the communication process and their riding status through the display output screen. Of course, the above is only an example to illustrate the effect, and this embodiment does not make any specific limitation to this.

[0109] In another example, in the process of adjusting the user communication screen according to the user position to obtain the display output screen, the voice text corresponding to the sub-audio information of each target user can also be combined to perform dialogue recognition to obtain dialogue relationship information corresponding to the target object. The dialogue relationship information can represent the dialogue relationship between the target users. For example, there are four target objects A, B, C, and D in the car who are having a dialogue. The dialogue relationship information obtained by performing dialogue recognition on the voice text can indicate that target object A is having a dialogue with target object D, and target object B is having a dialogue with target object C. At this time, when the user communication screen is adjusted according to the user position and the dialogue relationship information, the display output screen is obtained, so that the display output screen displayed on the display screen in front of target object A can only include the communication screen of target object D, the display output screen displayed on the display screen in front of target object D can only include the communication screen of target object A, the display output screen displayed on the display screen in front of target object B can only include the communication screen of target object C, and the display output screen displayed on the display screen in front of target object C can only include the communication screen of target object B. Of course, other display modes may also be used, and the user may display configuration information to select and configure the mode, that is, to adjust the user communication screen according to the user position to obtain a display output screen. This may also include obtaining display configuration information, and adjusting the user communication screen according to the display configuration information and the user position to obtain a display output screen, wherein the display configuration information may indicate the specific display mode of each display screen, and different display modes may display different screens as in the aforementioned examples, and this embodiment does not specifically limit this.

[0110] In an optional embodiment of the present application, after step S120 performs speech separation on the mixed audio information to obtain at least one sub-audio information, it may specifically include the following steps: extracting keywords from each sub-audio information to obtain user keywords; when the user keyword belongs to a preset control keyword, outputting a control execution prompt message.

[0111] In this embodiment, after performing speech separation on the mixed audio information to obtain at least one sub-audio information, keyword extraction can be performed on each sub-audio information to obtain user keywords, wherein the keyword extraction can refer to keywords used to represent specific content of the sub-audio information, rather than modal particles or conjunctions, etc.; thereby, semantic analysis can be performed on the user keywords to determine whether the user keywords belong to preset control keywords, which can represent keywords pre-configured for vehicle control, such as turning up the air conditioner, opening the windows, etc.; when the user keywords belong to the preset control keywords, it means that the user may have a need to control the vehicle, and control execution prompt information can be output at this time, wherein the control execution prompt information can represent prompt information used to prompt the user whether it is necessary to control the vehicle according to the user keywords, thereby avoiding the user from inadvertently directly controlling the vehicle during the vehicle communication process, resulting in a low user experience, and can remind the user whether it is necessary to control the vehicle through the control execution prompt information.

[0112] In one example, when the sub-audio information is "At that time at his house, I really wanted to turn up the air conditioner", the sub-audio information is subjected to keyword extraction to obtain the user keywords "home" and "air conditioner turn up", and it can be determined that the user keyword "home" does not belong to the preset control keyword and does not need to perform subsequent steps, while the user keyword "air conditioner turn up" belongs to the preset control keyword, and control execution prompt information can be output at this time, such as outputting the prompt information of "whether to turn up the air conditioner". Of course, the above is only an example for illustration, and this embodiment does not make specific limitations on this.

[0113] like Figure 3 As shown, the present application also discloses an embodiment, providing a vehicle speech recognition and processing device, including:

[0114] An acquisition module 310 is used to acquire mixed audio information and video information in the vehicle;

[0115] A speech separation module 320 is used to perform speech separation on the mixed audio information to obtain sub-audio information;

[0116] A determination module 330, configured to determine a target user corresponding to each piece of the sub-audio information according to the sub-audio information and / or the video information;

[0117] A user matching module 340 is used to perform user matching based on the target user and the video information to obtain image status information corresponding to the target user, where the image status information is the riding status of the target user in the car;

[0118] The generating module 350 is used to generate a user communication screen according to the image state information and the sub-audio information corresponding to the target user, and output and display the user communication screen.

[0119] In an optional embodiment of the present application, the determining module 330 may specifically include:

[0120] A first acquisition unit, configured to extract audio time information from the sub-audio information when the speech feature comparison result is a feature mismatch result;

[0121] A first extraction unit, configured to extract, from the video information, a target video frame that time-matches the audio time information;

[0122] The first determination unit is used to determine the target user corresponding to the sub-audio information by combining the user voice feature and the target video frame.

[0123] In an optional embodiment of the present application, the following may also be included:

[0124] A second extraction unit is used to extract audio time information from the sub-audio information when the speech feature comparison result is a feature mismatch result;

[0125] A third extraction unit, configured to extract, from the video information, a target video frame that time-matches the audio time information;

[0126] The second determination unit is used to determine the target user corresponding to the sub-audio information by combining the user voice feature and the target video frame.

[0127] In an optional embodiment of the present application, the user matching module 340 may specifically include:

[0128] A first identification unit is used to perform user identification on the video information to obtain an image user;

[0129] A matching unit, used to match the target user with the image user to obtain a matching result;

[0130] A fourth extraction unit, configured to extract user status information corresponding to the image user from the video information when the matching result is a user matching result;

[0131] The first generating unit is used to determine the user status information as the image status information corresponding to the target user.

[0132] In an optional embodiment of the present application, the matching unit may include:

[0133] A first acquisition subunit is used to acquire first physiological characteristic information corresponding to the target user and acquire second physiological characteristic information corresponding to the image user when the target user is a preset user;

[0134] A first matching subunit, configured to match the first physiological characteristic information with the second physiological characteristic information to obtain a physiological characteristic result;

[0135] A generating subunit is used to generate the matching result according to the physiological characteristic result.

[0136] In an optional embodiment of the present application, the matching unit may include:

[0137] A second acquisition subunit is used to acquire the first physiological characteristic information and the sub-audio information corresponding to the target user when the target user does not belong to the preset user;

[0138] A third acquisition subunit is used to acquire second physiological characteristic information and lip movement information corresponding to the image user according to the video information;

[0139] A second matching subunit, configured to match the first physiological characteristic information with the second physiological characteristic information to obtain a physiological characteristic result;

[0140] A third matching subunit is used to match the speech text corresponding to the sub-audio information with the lip movement information to obtain a lip shape matching result;

[0141] The third determining subunit is used to obtain the matching result according to the physiological feature result and the lip shape matching result.

[0142] In an optional embodiment of the present application, the generating module 350 may include:

[0143] A third determining unit, configured to determine a user position and a target display screen identifier corresponding to each of the target users;

[0144] An adjustment unit, used for adjusting the user communication screen according to the user position to obtain a display output screen;

[0145] The first output unit is configured to output the display output screen by identifying a target display screen corresponding to the target display screen.

[0146] In an optional embodiment of the present application, the in-vehicle speech recognition and processing device may further include:

[0147] An extraction module, used for extracting keywords from each of the sub-audio information to obtain user keywords;

[0148] The prompt module is used to output control execution prompt information when the user keyword belongs to a preset control keyword.

[0149] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0150] like Figure 4 As shown, an embodiment of the present application provides an electronic device, including a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440;

[0151] Memory 430, for storing computer programs;

[0152] In one embodiment of the present application, the processor 410 is used to execute the program stored in the memory 430 to implement the control method for recognition processing of in-vehicle voice provided by any of the aforementioned method embodiments, by acquiring mixed audio information and video information in the car, performing voice separation on the mixed audio information to obtain sub-audio information; then determining the target user corresponding to each sub-audio information according to the sub-audio information and / or video information; and based on the target user, performing user matching in combination with the video information to obtain image status information corresponding to the target user, where the image status information is the riding status of the target user in the car; generating a user communication screen according to the image status information and the sub-audio information corresponding to the target user, and then outputting the user communication screen for display; so that the in-vehicle users can obtain the communication content in real time and accurately through the user communication screen, so that the in-vehicle users can understand the in-vehicle communication situation in real time and accurately, thereby solving the problem of poor in-vehicle user communication experience caused by the inability of the in-vehicle users to accurately obtain the communication content in the existing related technology, and can effectively improve the in-vehicle user communication experience.

[0153] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the in-vehicle voice recognition and processing method provided by any of the aforementioned method embodiments are implemented, by acquiring mixed audio information and video information in the car, performing voice separation on the mixed audio information to obtain sub-audio information; then determining the target user corresponding to each sub-audio information according to the sub-audio information and / or video information; and based on the target user, user matching is performed in combination with the video information to obtain image status information corresponding to the target user, and the image status information is the riding status of the target user in the car; generating a user communication screen according to the image status information and the sub-audio information corresponding to the target user, and then outputting the user communication screen for display; so that the in-vehicle users can obtain the communication content in real time and accurately through the user communication screen, so that the in-vehicle users can understand the in-vehicle communication situation in real time and accurately, thereby solving the problem of poor in-vehicle user communication experience caused by the inability of the in-vehicle users to accurately obtain the communication content in the existing related technology, and can effectively improve the in-vehicle user communication experience.

[0154] An embodiment of the present application also provides a vehicle, which includes the in-vehicle speech recognition and processing device described in any of the aforementioned embodiments.

[0155] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0157] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0158] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for recognizing and processing in-vehicle speech, characterized in that: include: Obtain mixed audio and video information in the car; Performing voice separation on the mixed audio information to obtain sub-audio information; Determine a target user corresponding to each piece of the sub-audio information according to the sub-audio information and / or the video information; Based on the target user, user matching is performed in combination with the video information to obtain image state information corresponding to the target user, where the image state information is the riding state of the target user in the car; A user communication screen is generated according to the image state information and the sub-audio information corresponding to the target user, and the user communication screen is output and displayed.

2. The in-vehicle speech recognition and processing method according to claim 1, characterized in that: The determining, according to the sub-audio information and / or the video information, a target user corresponding to each piece of the sub-audio information includes: Obtaining a preset voice feature corresponding to at least one preset user; Extracting user voice features from each piece of sub-audio information; Comparing the user's voice features with various preset voice features to obtain a voice feature comparison result; In the case where the voice feature comparison result is a feature matching result, a preset user corresponding to the voice feature comparison result is determined as a target user. When the speech feature comparison result is a feature mismatch result, extracting audio time information from the sub-audio information; Extracting, from the video information, a target video frame that matches the audio time information in time; The target user corresponding to the sub-audio information is determined by combining the user voice feature and the target video frame.

3. The in-vehicle speech recognition and processing method according to claim 1, characterized in that: The performing user matching based on the target user and in combination with the video information to obtain image state information corresponding to the target user includes: Performing user identification on the video information to obtain an image user; Matching the target user with the image user to obtain a matching result; When the matching result is a user matching result, extracting user status information corresponding to the image user from the video information; The user status information is determined as the image status information corresponding to the target user.

4. The in-vehicle speech recognition and processing method according to claim 3, characterized in that: The matching the target user with the image user to obtain a matching result includes: In the case where the target user is a preset user, obtaining first physiological characteristic information corresponding to the target user, and obtaining second physiological characteristic information corresponding to the image user; Matching the first physiological characteristic information with the second physiological characteristic information to obtain a physiological characteristic result; The matching result is generated according to the physiological characteristic result.

5. The in-vehicle speech recognition and processing method according to claim 3, characterized in that: The matching the target user with the image user to obtain a matching result includes: When the target user does not belong to the preset user, obtaining the first physiological characteristic information and the sub-audio information corresponding to the target user; Acquire second physiological characteristic information and lip movement information corresponding to the image user according to the video information; Matching the first physiological characteristic information with the second physiological characteristic information to obtain a physiological characteristic result; Matching the speech text corresponding to the sub-audio information with the lip movement information to obtain a lip shape matching result; The matching result is obtained according to the physiological feature result and the lip shape matching result.

6. The in-vehicle speech recognition and processing method according to claim 1, characterized in that: Outputting and displaying the user communication screen includes: Determine a user location and a target display screen identifier corresponding to each of the target users; Adjusting the user communication screen according to the user position to obtain a display output screen; The target display screen corresponding to the target display screen identifier is used to output and display the display output screen.

7. The in-vehicle speech recognition and processing method according to claim 1, characterized in that: After performing voice separation on the mixed audio information to obtain a plurality of sub-audio information, the method further includes: Extracting keywords from each of the sub-audio information to obtain user keywords; In the case where the user keyword belongs to a preset control keyword, control execution prompt information is output.

8. A vehicle voice recognition and processing device, characterized in that: include: An acquisition module, used for acquiring mixed audio information and video information in the vehicle; A speech separation module, used for performing speech separation on the mixed audio information to obtain sub-audio information; a determination module, configured to determine a target user corresponding to each piece of the sub-audio information according to the sub-audio information and / or the video information; A user matching module, configured to perform user matching based on the target user and the video information, and obtain image status information corresponding to the target user, wherein the image status information is the riding status of the target user in the vehicle; A generating module is used to generate a user communication screen according to the image state information and the sub-audio information corresponding to the target user, and output and display the user communication screen.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the in-vehicle speech recognition and processing method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the in-vehicle speech recognition and processing method as described in any one of claims 1 to 7 is implemented.

Images